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Accountability  *  Integrity  *  Reliability 


United  States  General  Accounting  Office 
Washington,  DC  20548 


January  10, 2002 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  Warner 
Ranking  Minority  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Bob  Stump 
Chairman 

The  Honorable  Ike  Skelton 
Ranking  Minority  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Defense  Logistics  Agency  (DLA)  plays  a  critical  role  in  supporting 
America’s  military  forces  worldwide.  To  fulfill  this  role,  DLA  employs 
about  28,000  civilian  and  military  workers,  located  at  about  500  sites  in  all 
50  states  and  in  28  countries.  It  also  manages  about  4  million  supply  items 
and  processes  about  30  million  annual  supply  distribution  actions.  In  fiscal 
year  2000,  DLA  reported  that  these  operations  resulted  in  sales  to  the 
military  services  of  about  $13  billion.  DLA  relies  on  software-intensive 
systems  to  support  this  work.  An  important  determinant  of  the  quality  of 
software-intensive  systems,  and  thus  DLA’s  mission  performance,  is  the 
quality  of  the  processes  used  to  acquire  these  systems. 

This  report  is  one  in  a  series  of  products  to  satisfy  our  mandate  under  the 
fiscal  year  2001  Defense  Authorization  Act.'  The  act  directed  that  we 
review  DLA’s  efficiency  and  effectiveness  in  meeting  requirements,  its 
application  of  best  business  practices,  and  opportunities  for  improving  its 
operations.  As  agreed  with  your  offices,  the  objectives  of  this  review  of 
DLA’s  information  technology  (IT)  management  were  to  determine  (1) 
whether  DLA  has  the  effective  software  acquisition  processes  that  are 
necessary  to  modernize  and  maintain  systems  and  (2)  what  actions  DLA 
has  planned  or  in  place  to  improve  these  processes. 


‘Floyd  D.  Spence  National  Defense  Authorization  Act  for  Fiscal  Year  2001,  P.L.  106-398 
app.,  section  917. 
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Carnegie  Mellon  University’s  Software  Engineering  Institute  (SEI), 
recognized  for  its  expertise  in  software  processes,  has  developed  models 
and  methods  that  define  and  determine  organizations’  software  process 
maturity.  Together,  these  models  and  methods  provide  (1)  a  logical 
framework  for  baselining  an  organization’s  current  process  capabilities 
(i.e.,  determining  what  practices  are  effectively  implemented  [strengths], 
not  effectively  implemented  [weaknesses],  or  contain  mixed  or 
inconclusive  evidence  [observations])  and  (2)  a  structured  plan  for 
incremental  process  improvement.  These  models  and  methods  are 
generally  recognized  as  best  business  practices. 

Using  SEFs  Software  Acquisition  Capability  Maturity  Model^^  (SA-CMM  y 
and  SEl’s  software  capability  evaluation  method,  our  staff  (trained  at  SEI) 
evaluated  DLA’s  software  acquisition  maturity  in  six  of  seven  key  process 
areas  that  are  necessary  to  attain  a  “repeatable”  level  of  process  maturity.^ 
The  repeatable  level  of  process  maturity  is  level  2  on  SEI’s  five-level  scale. 
An  organization  at  the  repeatable  level  of  process  maturity  has  the 
necessary  process  discipline  in  place  to  repeat  earlier  successes  on  similar 
projects.  Organizations  that  do  not  satisfy  the  requirements  for  the 
repeatable  level  are  by  default  judged  to  be  at  level  1,  the  “initial”  level  of 
maturity.  This  means  that  their  processes  are  immature,  ad  hoc,  and 
sometimes  even  chaotic,  with  few  of  the  processes  defined  and  success 
dependent  mainly  on  the  heroic  efforts  of  individuals.  We  also  evaluated 
DLA  on  one  level-3,  or  “defined”  level,  process— acquisition  risk 
management.  We  included  acquisition  risk  management  because  many 
software  experts  consider  it  to  be  one  of  the  most  important  process 
areas. 

Our  evaluation  included  DLA’s  only  ongoing  software/system  acquisitions: 
the  Business  Systems  Modernization  (BSM)  and  the  Fuels  Automated 
System  (FAS).  Details  on  our  objectives,  scope,  and  methodology  are 


^Capability  Maturity  Model®”  is  the  service  mark  of  Carnegie  MeUon  University,  and  CMM 
is  registered  with  the  U.S.  Patent  and  Trademark  Office.  GAO  used  the  Softw^e 
Acquisition  Capability  Maturity  Model®”  Version  1.2  (CMU/SEI-99-TR-002,  April  1999),  the 
latest  version  of  the  model. 

*^The  six  key  process  areas  that  we  evaluated  are  software  acquisition  planning,  solicitation, 
requirements  development  and  management,  project  management,  contract  tracking  and 
oversight,  and  evaluation.  We  did  not  evaluate  DLA  against  the  seventh  key  process  area, 
transition  to  support,  because  the  contractors  who  are  implementing  the  systems  we 
evaluated  will  also  support  the  systems  when  they  are  operational,  rendering  transition  to 
support  irrelevant  for  these  acquisitions. 
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contained  in  appendix  I.  The  Department  of  Defense  (DOD)  provided  us 
with  comments  on  a  draft  of  this  report,  which  are  discussed  in  the 
“Agency  Comments”  section. 


Results  in  Brief 


DLA  does  not  have  mature  software  acquisition  processes  across  the 
agency,  as  evidenced  by  the  wide  disparity  in  the  rigor  and  discipline  of 
processes  between  the  two  systems  we  evaluated.  Whereas  BSM  fully 
satisfied  requirements  for  most  of  the  key  process  areas  evaluated,  FAS 
did  not  fully  satisfy  all  the  criteria  for  any  key  process  area.  More 
specifically,  BSM  satisfied  all  requirements  for  three  level-2  key  process 
areas — software  acquisition  management,  project  management,  and 
contract  tracking  and  oversight — and  for  the  one  level-3  key  process  area 
that  we  evaluated — ^acquisition  risk  management.  Further,  BSM  satisfied 
all  but  a  few  practices  in  the  other  level-2  key  process  areas — solicitation, 
requirements  development  and  management,  and  evaluation.  On  the  other 
hand,  FAS  did  not  fully  satisfy  all  requirements  for  any  of  the  level-2  key 
process  areas,  and  also  did  not  satisfy  the  one  level-3  key  process  area  we 
evaluated.  According  to  DLA  officials,  the  variance  between  BSM  and  FAS 
software  acquisition  maturity  can  be  attributed  in  part  to  differences  in  the 
level  of  resources  that  each  project  committed  to  acquisition  process 
controls.  This  means  that  DLA  does  not  have  effective  corporate  processes 
for  consistently  acquiring  software  (the  most  costly  and  complex 
component  of  systems),  which  can  lead  to  the  acquisition  of  systems  that 
do  not  meet  the  information  needs  of  management  and  staff,  do  not 
provide  support  for  necessary  programs  and  operations,  and  cost  more 
and  take  longer  than  expected  to  complete. 

Moreover,  DLA  does  not  have  a  software  process  improvement  program  in 
place  to  effectively  strengthen  its  corporate  software  acquisition 
processes.  Earlier  this  year,  we  reported  that  DLA  does  not  have  a 
software  process  improvement  program,  having  eliminated  the  program  in 
1998.'  We  also  reported  that  DLA’s  Chief  Information  Officer  (CIO)  stated 
that  the  program  would  be  reestablished.  However,  DLA  stiU  does  not 
have  written  plans  and  milestones  for  doing  so  because  the  improvement 
program  has  not  been  an  agency  priority.  Without  a  software  process 
improvement  program,  it  is  xmlikely  that  DLA  can  effectively  Improve  its 
institutional  software  acquisition  capabilities,  which  in  turn  means  that 


^DOD  Information  Technology:  Software  and  Systems  Process  Improvement  Programs 
Vary  in  Use  of  Best  Practices  (GAO-01-1 16,  March  30, 2001). 
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DLA’s  software  projects  will  be  at  risk  of  not  delivering  promised 
capabilities  on  time  and  within  budget. 

To  reduce  DLA’s  software  acquisition  project  risks,  we  are  recommending 
actions  aimed  at  (1)  correcting  BSM  and  FAS  process  weaknesses  and  (2) 
establishing  a  framework  for  long-term  institution  software  process 
improvement. 

DOD  provided  what  it  termed  “official  oral  comments”  on  a  draft  of  this 
report.  In  its  comments,  DOD  stated  that  it  generally  concurred  with  the 
report  and  that  it  concurred  with  the  recommendations. 


Background 


DLA  is  the  Department  of  Defense’s  (DOD)  logistics  manager  for  all  DOD 
consumable  items^  and  some  department  repair  items.®  Its  primary 
business  function  is  to  provide  supply  support  in  order  to  sustain  mflitary 
operations  and  readiness.  In  addition  to  this  primary  function,  which  DLA 
refers  to  as  either  “materiel  management”  or  “supply-chain  management,” 
DLA  performs  five  other  major  business  functions:  distributing  materiel 
ordered  from  its  inventory;  purchasing  fuels  for  DOD  and  the  U.S. 
government;  storing  strategic  materiel;^  marketing  surplus  DOD  materiel 
for  reuse  and  disposal;  and  providing  numerous  information  services,  such 
as  item  cataloging,®  for  DOD,  the  United  States,  and  selected  foreign 
governments.  DLA  consists  of  a  central  command  authority  supported  by  a 
munber  of  field  commands  that  manage  the  agency’s  six  business 
functions. 

Until  about  1997,  DLA  generally  developed  its  systems  in-house.  Since 
then,  the  agency  has  begun  to  acquire  systems,  relying  on  contractors  for 
system  development  and  managing  the  acquisition  of  these  systems. 
Currently,  DLA  is  in  the  process  of  acquiring  two  systems:  Business 
Systems  Modernization  (BSM)  and  Fuels  Automated  System  (FAS). 


^Consumable  items  include  such  commodities  as  subsistence  (food),  fuels,  medical 
supplies,  clothing,  and  construction  equipment. 

®These  repair  items  are  spare  and  repair  parts  that  support  about  1,400  DOD  weapons 
systems.  Each  of  the  military  services  also  manages  its  own  service-uraque  repair  items. 

^“Strategic  materiel”  is  defined  as  any  item  needed  to  sustain  the  United  States  in  the  event 
of  a  national  emergency. 

®DLA  defines  “item  cataloging”  to  include  all  activities  that  describe  the  technical 
characteristics  and  data  for  an  individual  item  of  supply. 
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•  BSM  is  intended  to  modernize  DLA’s  materiel  management  business 
function,  changing  the  agency  from  being  solely  a  provider  and  manager  of 
physical  inventory  to  being  a  manager  of  supply  chains.  In  this  role,  DIA 
would  link  customers  with  appropriate  suppliers  and  track  physical  and 
financial  business  practices.  It  is  planning  to  replace  two  large  legacy 
systems,  as  weU  as  several  supporting  programs,  that  are  more  than  30 
years  old  and  are  not  integrated.  BSM  is  based  on  commercially  available 
software  products.  DLA  plans  to  acquire  and  deploy  its  BSM  system 
solution  through  a  series  of  four  system  releases/increments.  First,  it  plans 
to  demonstrate  successfid  application  of  its  new  concept  of  doing  business 
for  selected  commodities — ^namely,  earth-moving  equipment, 
medical/pharmaceutical  supplies,  and  F/A-18  engine  components — at  the 
three  Defense  Supply  Centers.  If  this  first  release  is  successfully 
demonstrated,  DLA  plans  to  expand  the  system  solution  to  other 
commodities  in  three  additional  increments.  DIA  plans  to  invest 
approximately  $668  milUon  to  acquire  and  implement  BSM  from  fiscal 
years  2000  through  2005. 

•  FAS  is  intended  to  help  the  Defense  Energy  Support  Center  manage  about 
$5  billion  in  contracts  with  petroleum  suppliers  each  year.  FAS  is  to  be  a 
multifunctional  system  that  provides  for,  among  other  things,  point-of-sale 
data  collection  inventory  control,  finance  and  accounting,  procurement, 
and  facilities  management.  FAS,  which  relies  on  a  commercially  available 
software  package,  is  being  fielded  incrementally.  Increment  1  is  the  base- 
level  operational  module  that  is  currently  being  deployed  to  base-level 
sites  worldwide.  The  second  increment  is  the  enterprise-level  system, 
which  is  to  be  deployed  to  its  direct  delivery  commodity  business  unit. 

DLA  plans  to  invest  $293  miUion  in  FAS  from  fiscal  year  1995  through 

2002. 

SEI’s  SA-CMM  is  used  to  measure  an  organization’s  capability  to  manage 
the  acquisition  of  software.  SEI’s  expertise  in,  and  model  and  methods  for, 
determining  software  process  assessment  are  recognized  and  accepted 
throughout  the  software  industry.  The  model  defines  five  levels  of 
software  acquisition  maturity.  Each  level  of  maturity  (except  level  1) 
indicates  process  capability  in  relation  to  key  process  areas.  For  a 
maturity  level  to  be  achieved,  aU  key  process  areas  related  to  that  level 
must  be  implemented  effectively. 

The  second  level  of  process  maturity,  level  2  (referred  to  as  the  repeatable 
level),  demonstrates  that  basic  management  processes  are  established  to 
track  performance,  cost,  and  schedule,  and  the  necessary  discipline  is  in 
place  to  repeat  earlier  successes  on  similar  projects.  Organizations  that  do 
not  effectively  implement  aU  key  process  areas  for  the  repeatable  level  are, 
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by  default,  at  level  1,  the  initial  level  of  maturity.  Level-1  processes  can  be 
described  as  immature,  ad  hoc,  and  sometimes  chaotic;  success  in 
software  acquisition  for  these  organizations  depends  on  the  abUity  and 
commitment  of  the  staff  involved.  Figure  1  further  e3q)lains  the  five-level 
software  acquisition  model. 


We  evaluated  DLA  against  six  of  the  seven  level-2  (repeatable)  key  process 
areas  in  the  SA-CMM.  We  did  not  evaluate  DLA  on  the  seventh  key  process 
area — ^transition  to  support — ^because  the  contractors  who  are 
implementing  BSM  and  FAS  will  support  these  systems  when  they  are 
operational,  rendering  transition  to  support  irrelevant  for  these 
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acqiiisitions.  We  evaluated  DLA  against  one  level-3  (defined)  key  process 
area — acquisition  risk  management — ^because  many  software  acquisition 
experts  consider  it  to  be  one  of  the  most  important  key  process  areas. 
These  key  process  areas  are  described  in  table  1. 

Table  1:  Six  SA-CMM  Level-2  and  One  Level-3  Key  Process  Areas 

Key  process 
areas 

Description 

Examples  of  SA-CMM  required  practices* 

SA-CMM  level  2  _ _ _ 

Software 

acquisition 

planning 

Ensuring  that  reasonable  planning 
for  the  software  acquisition  is 
conducted  and  that  all  elements  of 
the  project  are  included. 

Includes  (1)  having  a  written  software  acquisition  policy,  (2)  having 
adequate  resources  for  software  acquisition  planning,  (3)  developing  and 
documenting  the  software  acquisition  strategy  and  plan,  (4)  having 
management  review  software  acquisition  planning  activities,  and  (5) 
making  and  using  measurements  to  determine  the  status  of  software 
acquisition  planninq  activities. 

Solicitation 

Ensuring  that  award  is  made  to  the 
contractor  most  capable  of 
satisfying  the  specified 
requirements. 

Includes  (1)  designating  responsibility  for  the  software  portion  of  the 
solicitation,  (2)  preparing  cost  and  schedule  estimates  for  the  software 
products  and  services  being  acquired,  (3)  having  a  written  policy  for  the 
conduct  of  the  software  portion  of  the  solicitation,  and  (4)  having  an 
independent  review  of  cost  and  schedule  estimates  for  the  software 
products  and  services  being  acquired. 

Requirements 
development  and 
management 

Establishing  a  common  and 
unambiguous  definition  of  software 
acquisition  requirements  that  Is 
understood  by  the  acquisition  team, 
system  users,  and  contractor(s). 

This  key  process  area  involves  two 
subprocesses:  (1)  developing  a 
baseline  set  of  software-related 
contractual  requirements  and  (2) 
managing  these  requirements  and 
changes  to  these  requirements  for 
the  duration  of  the  acquisition. 

Includes  (1)  having  a  written  policy  for  managing  the  software-related 
contractual  requirements,  (2)  having  a  group  that  is  responsible  for 
performing  requirements  development  and  management  activities,  (3) 
ensuring  that  the  team  performs  its  activities  in  accordance  with  Its 
documented  requirements  development  and  management  plans,  (4) 
appraising  system  requirements  change  requests  for  their  impact  on  the 
software  being  acquired,  (5)  appraising  changes  to  the  software-related 
contractual  requirements  for  their  impact  on  performance  and  contract 
schedule  and  cost,  and  (6)  measuring  and  reporting  on  the  status  of 
requirements  development  and  management  activities  to  management. 

Project 

management 

Managing  the  activities  of  the 
project  office  and  supporting 
contractor(s)  to  ensure  a  timely, 
efficient,  and  effective  software 
acquisition. 

Includes  (1 )  designating  responsibility  for  project  management,  (2)  having 
a  written  policy  for  the  management  of  the  software  project,  (3)  haying 
adequate  resources  for  the  duration  of  the  software  acquisition  project, 

(4)  documenting  the  roles,  responsibilities,  and  authority  for  the  project 
functions,  (5)  tracking  the  risks  associated  with  cost,  schedule,  and 
resources,  and  (6)  using  a  corrective  action  system  for  identifying, 
recordinq,  trackinq,  and  correctinq  problems. 
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Key  process 
areas 

Description 

Examples  of  SA-CMM  required  practices* 

Contract  tracking 
and  oversight 

Ensuring  that  the  software  activities 
under  contract  are  being  performed 
in  accordance  with  contract 
requirements  and  that  products  and 
services  will  satisfy  contract 
requirements. 

Includes  (1)  designating  responsibility  for  contract  tracking  and  oversight, 
(2)  including  contract  specialists  in  the  project  team,  (3)  ensuring  that 
individuals  performing  contract  tracking  and  oversight  activities  have 
experience  or  receive  training,  (4)  having  a  documented  plan  for  contract 
tracking  and  oversight,  and  (5)  comparing  the  actual  cost  and  schedule  of 
the  contractor’s  software  engineering  effort  to  planned  schedules  and 
budgets. 

Evaluation 

Determining  that  the  acquired 
software  products  and  services 
satisfy  contract  requirements  before 
acceptance. 

Includes  (1)  designating  responsibility  for  planning,  managing,  and 
performing  evaluation  activities,  (2)  ensuring  that  adequate  resources  are 
provided  for  evaluation  activities,  (3)  documenting  evaluation  plans  and 
conducting  evaluation  activities  in  accordance  with  the  plan,  (4) 
developing  and  managing  evaluation  requirements  in  conjunction  with 
developing  software  technical  requirements,  and  (5)  measuring  and 
reoortina  on  the  status  of  evaluation  activities  to  management. 

SA-CMM  level  3 

Acquisition  risk 
management 

Identifying  risks  as  early  as  possible 
and  adjusting  the  acquisition  to 
mitigate  those  risks. 

Includes  (1)  having  a  written  policy  for  managing  software  acquisition  risk, 
(2)  designating  responsibility  for  software  acquisition  risk  activities,  (3) 
providing  adequate  resources  for  software  acquisition  risk  management 
activities,  (4)  developing  a  software  acquisition  risk  management  plan, 
and  (5)  measuring  and  reporting  on  the  status  of  acquisition  risk 
management  activities  to  management. 

“We  included  only  examples  of  the  SA-CMM  key  practices. 


Source:  GAO,  based  on  SEI  data. 


As  established  by  the  model,  each  key  process  area  contains  five  common 
features — commitment  to  perform,  ability  to  perform,  activities  to  be 
performed,  measurement  and  analysis  of  activities,  and  verification  of 
activities’  implementation.  These  five  features  collectively  provide  a 
framework  for  the  implementation  and  institutionalization  of  the  key 
process  areas.  The  common  feature  definitions  are  as  follows: 

•  Commitment  to  perform:  This  feature  describes  the  actions  that  the 
organization  takes  to  establish  the  process  and  ensure  that  it  can  endure. 
Key  practices  typically  involve  establishing  organizational  policies  and 
sponsorship. 

•  Ability  to  perform:  This  feature  describes  the  preconditions  that  must 
exist  in  the  project  or  organization  to  implement  the  software  acquisition 
process  competently.  Key  practices  typically  include  assigning 
responsibility  and  providing  training. 

•  Activities  to  be  peTformed:  This  feature  describes  the  roles  and 
procedures  necessary  to  implement  a  key  process  area.  Key  practices 
typically  involve  establishing  plans  and  procedures,  performing  the  work, 
tracking  it,  and  taking  appropriate  management  actions. 

•  Measurement  and  analysis  of  activities:  This  feature  describes  the  steps 
necessary  to  measure  progress  and  analyze  the  measurements.  Key 
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practices  typically  involve  defining  the  measurements  to  be  taken  and  the 
analyses  to  be  conducted  to  determine  the  status  and  effectiveness  of  the 
activities  performed. 

•  Verification  of  activities'  implementation:  This  feature  describes  the 
steps  the  organization  must  take  to  ensure  that  project  activities  are 
performed  in  accordance  with  established  processes.  Key  practices 
typically  involve  regular  reviews  by  management. 

Each  common  feature  consists  of  a  number  of  key  practices — specific 
actions  such  as  developing  an  organizational  policy  for  software 
acquisition,  developing  various  plans  for  software  acquisition  activities, 
and  tracking  a  contractor’s  progress.  When  an  organization  is  evaluated 
against  the  SA-CMM,  comparisons  of  actual  performance  against  a  key 
practice  can  result  in  one  of  four  possible  outcomes  or  ratings: 

•  Strength:  The  key  practice  involved  was  effectively  implemented. 

•  Weakness:  The  key  practice  was  not  effectively  implemented  or  was  not 
implemented. 

•  Observation:  The  key  practice  was  evaluated,  but  cannot  be  characterized 
as  a  strength  because  (1)  the  project  team  did  not  provide  sufficient 
evidence  to  support  a  strength  rating  or  (2)  the  key  practice  was  only 
partially  performed. 

•  Not  rated:  The  key  practice  is  not  relevant  to  the  project. 

To  achieve  the  repeatable  level,  DLA  would  have  to  demonstrate  that  the 
key  practices  related  to  this  level  were  implemented  effectively  in  the 
software  acquisition  projects  being  evaluated,  and  thus  the  project 
successes  can  be  repeated  in  future  projects. 


DLA  Lacks  the 
Capability 
to  Acquire  Software 
Effectively 


DLA  is  not  at  level  2  (the  repeatable  level  of  maturity)  when  compared 
with  the  SA-CMM— meaning  that  DLA  does  not  possess  an  agencywide  or 
corporate  ability  to  effectively  acquire  software-intensive  systems. 
Whereas  DIA’s  BSM  project  fully  or  substantially  satisfied  SEI’s  SA-CMM 
requirements  for  the  key  process  areas  for  level  2,  as  well  as  requirements 
for  one  level  3  (defined  level)  key  process  area,  its  FAS  project  did  not 
satisfy  all  the  criteria  for  any  of  these  key  process  areas.  A  discussion  of 
how  each  system  compared  with  the  SA-CMM  is  summarized  below. 
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BSM  Satisfied  or 
Substantially  Satisfied 
All  Key  Process  Areas 


BSM  completely  satisfied  requirements  for  three  of  the  level-2  key  process 
areas,  as  well  as  for  the  one  level-3  key  process  area,  and  substantially 
satisfied  requirements  for  the  remaining  three  level-2  key  process  areas 
that  we  evaluated.®  (See  table  2  for  the  percentage  of  strengths  and 
weakness  for  each  area  evaluated.)  According  to  BSM  officials,  satisfying 
the  criteria  for  the  key  process  areas  is  attributable  to  the  following 
factors:  allocating  adequate  resources;  following  good  program 
management  practices,  as  defined  in  DOD  Directive  5000;  and  working 
closely  with  relevant  oversight  groups.  To  address  those  few  weaknesses 
that  we  identified,  project  officials  told  us  that  they  have  initiated 
corrective  action. 


Table  2:  Key  Process  Area  Strengths  and  Weaknesses  for  BSM 

Strengths  Weaknesses 

Observations 

Key  process  area 

_ (%i _ 

(%) 

Software  acquisition  planning 

100 

0 

0 

Solicitation 

94 

6 

0 

Requirements  development  and 
management 

79 

21 

0 

Project  management 

100 

0 

0 

Contract  tracking  and  oversight 

100 

0 

0 

Evaluation 

93 

0 

7 

Acquisition  risk  management 

100 

0 

0 

Source:  GAO  calculations,  based  on  data  and  inten/iews  with  Business  Systems  Modernization 
officials. 


BSM  satisfied  all  key  practices  in 

•  software  acquisition  planning,  such  as  (1)  having  a  written  software 
acquisition  policy,  (2)  having  adequate  resources  for  software  acquisition 
planning  activities,  (3)  developing  and  documenting  the  software 
acquisition  strategy  and  plan,  and  (4)  making  and  using  measurements  to 
determine  the  status  of  software  acquisition  planning  activities. 

•  project  management,  including  (1)  designating  responsibility  for  project 
management,  (2)  having  a  written  policy  for  the  management  of  the 
software  project,  (3)  having  adequate  resources  for  the  duration  of  the 


®We  did  not  evaluate  BSM  against  the  transition-to^support  key  process  area  because  the 
contractor  who  is  implementing  BSM  will  also  support  this  system  when  it  is  operational, 
rendering  transition  to  support  irrelevant. 
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software  acquisition  project,  and  (4)  tracking  the  risks  associated  with 
cost,  schedule,  resources,  and  the  technical  aspects  of  the  project. 

•  contract  tracking  and  oversight,  including  (1)  designating  responsibility 
for  contract  tracking  and  oversight,  (2)  including  contract  specialists  in 
the  project  team,  and  (3)  having  a  documented  plan  for  contract  tracking 
and  oversight. 

•  acquisition  risk  management,  such  as  (1)  having  a  risk  management  plan, 
(2)  having  a  written  policy  for  the  management  of  software  acquisition 
risk,  and  (3)  measuring  and  reporting  on  the  status  of  acquisition  risk 
management  activities  to  management. 

BSM  also  satisfied  all  but  one  key  practice  in  solicitation.  Strengths 
included  (1)  designating  responsibility  for  the  software  portion  of  the 
solicitation,  (2)  preparing  cost  and  schedule  estimates  for  the  software 
products  and  services  being  acquired,  and  (3)  having  an  independent 
review  of  cost  and  schedule  estimates  for  the  software  products  and 
services  being  acquired.  BSM’s  one  weakness  in  this  key  process  area  was 
in  not  having  a  written  policy  for  the  software  portion  of  the  solicitation. 
This  is  significant  because,  according  to  the  SEI,  an  institutional  policy 
provides  for  establishing  an  enduring  process. 

BSM  also  satisfied  all  but  three  key  practices  in  requirements  development 
and  management.  Strengths  included  (1)  having  a  written  policy  for 
managing  the  software-related  contractual  requirements,  (2)  having  a 
group  that  is  responsible  for  performing  requirements  development  and 
management  activities,  and  (3)  measuring  and  reporting  to  management 
on  the  status  of  requirements  development  and  management  activities. 

One  of  the  three  weaknesses  was  the  lack  of  a  documented  requirements 
development  and  management  plan.  Such  a  plan  provides  a  roadmap  for 
completing  important  requirements  development  and  management 
activities.  Without  it,  projects  risk  either  not  performing  important  tasks  or 
not  performing  them  effectively.  The  other  two  weaknesses  involved  the 
project  office’s  appraisal  of  system  requirements  changes.  Specifically, 
BSM  did  not  appraise  (1)  requests  to  change  system  requirements  for  their 
impact  on  the  software  being  acquired  or  (2)  all  changes  to  the 
requirements  for  impact  on  performance  and  contract  schedule  and  cost. 
These  activities  are  critical  to  making  informed,  risk-based  decisions  about 
whether  to  approve  requirements  changes. 

Last,  BSM  satisfied  all  but  one  key  practice  in  evaluation,  and  we  do  not 
view  that  practice  as  significant.  Strengths  included  (1)  designating 
responsibility  for  contract  tracking  and  oversight,  (2)  documenting 
evaluation  plans  and  conducting  evaluation  activities  in  accordance  with 
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FAS  Did  Not  Satisfy  Any 
of  the  Key  Process  Areas 


the  plan,  and  (3)  developing  and  managing  evaluation  requirements  in 
conjunction  with  developing  software  technical  requirements. 

By  generally  satisfying  these  key  process  areas  for  its  BSM  project,  DLA 
has  increased  the  chances  that  the  software  acquired  on  this  project  will 
meet  stated  requirements  and  will  be  delivered  on  time  and  within  budget. 

See  appendix  II  for  more  detailed  information  on  key  process  areas  and 
our  findings  on  BSM. 


Because  of  the  number  and  severity  of  its  key  practice  weaknesses,  FAS 
did  not  fully  satisfy  all  the  criteria  for  any  of  the  five  level-2  SA-CMM  key 
process  areas  or  for  the  one  level-3  key  process  area  that  we  evaluated.*'' 
(See  table  3  for  the  percentage  of  stren^s  and  weakness  for  each  area 
evaluated.)  According  to  FAS  officials,  these  weaknesses  are  attributable 
to  a  lack  of  adequate  resources  for  the  process  areas.  However,  these 
officials  stated  that  they  are  currently  in  the  process  of  reorganizing  and 
addressing  resource  shortages. 


Table  3:  Key  Process  Area  Strengths  and  Weaknesses  for  FAS 

Kev  process  area 

Strengths 

(%) 

Weaknesses  Observations 

_ 

Not 

rated 

(%) 

Software  acquisition  planning 

80 

13 

7 

- 

Requirements  development  and 
management 

43 

43 

14 

Project  management 

63 

37 

- 

- 

Contract  tracking  and  oversight 

65 

29 

6 

Evaluation 

60 

13 

13 

14 

Acquisition  risk  management 

20 

73 

7 

— 

Source:  GAO  calculations,  based  on  data  and  interviews  with  Fuels  Automated  System  officials. 


In  the  software-acquisition-planning  key  process  area,  FAS  had  12 
strengths,  2  weaknesses,  and  1  observation.  Strengths  included,  among 
other  things,  (1)  having  a  written  software  acquisition  policy,  (2) 
developing  and  documenting  the  software  acquisition  strategy  and  plan. 


“'We  did  not  evaluate  FAS  on  solicitation  because  it  was  a  sole-source  purchase,  or  on 
transition  to  support  because  the  contractor  who  is  implementing  FAS  will  also  support 
this  system  when  it  is  operational,  rendering  transition  to  support  irrelevant. 
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and  (3)  having  management  review  software-acquisition-planning 
activities.  Weaknesses  included  (1)  not  having  adequate  resources  for 
software-acquisition-planning  activities  and  (2)  not  measuring  the  status 
of  the  software-acquisition-planning  activities  and  resultant  products.  The 
weaknesses  are  significant  because  they  could  prevent  management  from 
developing  effective  plans,  from  being  aware  of  problems  in  meeting 
planned  commitments,  or  from  taking  necessary  corrective  actions 
expeditiously. 

In  the  requirements  development  and  management  key  process  area,  FAS 
had  six  strengths,  six  weaknesses,  and  two  observations.  Examples  of 
strengths  included  (1)  having  a  written  policy  for  managing  the  software- 
related  contractual  requirements  and  (2)  having  a  group  that  is  responsible 
for  performing  requirements  development  and  management  activities. 
However,  we  found  weaknesses  in  important  key  practices  that  jeopardize 
effective  control  of  the  requirements  baseline  and  can  result  in  software 
products  that  do  not  meet  cost,  schedule,  or  performance  objectives. 
Specific  examples  of  weaknesses  included  (1)  not  having  a  documented 
requirements  development  and  management  plan,  (2)  not  appraising 
requests  to  change  system  requirements  for  their  impact  on  the  software 
being  acquired,  (3)  not  appraising  changes  to  the  software-related 
contractual  requirements  for  their  impact  on  performance  and  contract 
schedule  and  cost,  and  (4)  not  measuring  and  reporting  to  management  on 
the  status  of  requirements  development  and  management  activities. 

In  the  project  management  key  process  area,  FAS  had  10  strengths  and  6 
weaknesses.  Strengths  included,  among  other  things,  (1)  designating 
responsibility  for  project  management,  (2)  having  a  written  policy  for  the 
management  of  the  software  project,  and  (3)  using  a  corrective  action 
system  for  identifying,  recording,  tracking,  and  correcting  problems. 
Examples  of  weaknesses  included  (1)  not  having  adequate  resources  for 
the  duration  of  the  software  acquisition  project,  (2)  not  documenting  the 
roles,  responsibilities,  and  authority  for  the  project  functions,  and  (3)  not 
tracking  the  risks  associated  with  cost,  schedule,  and  resources.  These 
weaknesses  are  significant  because  they  coiild  jeopardize  the  project’s 
ability  to  ensure  that  important  project  management  and  contractor 
activities  are  defined,  understood,  and  completed. 

FAS  had  11  strengths,  5  weaknesses,  and  1  observation  in  the  contract 
tracking  and  oversight  key  process  area.  Strengths  included,  among  other 
things,  (1)  designating  responsibility  for  contract  tracking  and  oversight, 
(2)  including  contract  specialists  on  the  project  team,  and  (3)  ensuring 
that  individuals  performing  contract  tracking  and  oversight  activities  had 
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experience  or  received  training.  Examples  of  weaknesses  included  (1)  not 
having  a  documented  plan  for  contract  tracking  and  oversight  and  (2)  not 
comparing  the  actual  cost  and  schedule  of  the  contractor’s  software 
engineering  effort  with  planned  schedules  and  budgets.  Because  of  these 
weaknesses,  FAS  contractor  tracking  and  oversight  activities  are 
undisciplined  and  unstructured,  thereby  increasing  the  chances  of  FAS 
software  acquisitions  being  late,  costing  more  than  expected,  and  not 
performing  as  intended. 

In  the  evaluation  key  process  area,  FAS  had  nine  strengths,  two 
weaknesses,  two  observations,  and  two  areas  that  were  not  rated. 
Strengths  included,  among  other  things,  (1)  designating  responsibility  for 
planning,  managing,  and  performing  evaluation  activities,  (2)  documenting 
evaluation  plans  and  conducting  evaluation  activities  in  accordance  with 
the  plan,  and  (3)  developing  and  managing  evaluation  requirements  in 
conjunction  with  developing  software  technical  requirements.  Weaknesses 
were  (1)  not  ensuring  that  adequate  resources  were  provided  for 
evaluating  activities  and  (2)  not  measuring  and  reporting  on  the  status  of 
evaluation  activities  to  management.  These  weaknesses  are  significant 
because  they  preclude  DLA  decisionmakers  fi:om  knowing  whether 
contractor-developed  software  is  meeting  defined  requirements. 

FAS  performed  poorly  in  the  one  level-3  key  process  area  that  we 
evaluated — acquisition  risk  management — ^with  3  strengths,  11 
weaknesses,  and  1  observation.  Examples  of  strengths  included  (1)  having 
a  written  policy  for  the  management  of  software  acquisition  risk  and  (2) 
designating  responsibility  for  software  acquisition  risk  activities. 
Weaknesses  included,  among  others,  (1)  not  having  adequate  resources  for 
performing  risk  management  activities,  (2)  not  having  a  software  risk 
management  plan,  and  (3)  not  measuring  and  reporting  on  the  status  of 
acquisition  risk  management  activities  to  management.  Because  of  these 
weaknesses,  the  project  office  does  not  have  adequate  assurance  that  it 
will  promptly  identify  risks  and  effectively  mitigate  them  before  they 
become  problems. 

By  not  satisfying  any  of  these  key  process  areas  for  its  FAS  project,  DLA  is 
unnecessarily  increasing  the  risk  that  the  software  acquired  on  this  project 
wiU  not  meet  stated  requirements  and  will  not  be  delivered  on  time  and 
within  budget. 

Appendix  III  provides  more  details  on  the  key  process  areas  and  our 
findings  on  FAS. 
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DLA  Lacks  Effective 
Software  Process 
Improvement 


The  quality  of  the  processes  involved  in  developing,  acquiring,  and 
engineering  software  and  systems  has  a  significant  effect  on  the  quality  of 
the  resulting  products.  Accordingly,  process  improvement  programs  can 
increase  product  quality  and  decrease  product  costs.  Public  and  private 
organizations  have  reported  significant  returns  on  investment  through 
such  process  improvement  programs.  In  particular,  SEI  has  published 
reports  of  benefits  realized  through  process  improvement  programs.  For 
example,  SEI  reported  in  1995“  that  a  major  defense  contractor  had 
implemented  a  process  improvement  program  in  1988  and  by  1995  had 
reduced  its  re-work  costs  from  about  40  percent  of  project  cost  to  about 
10  percent,  increased  staff  productivity  by  about  170  percent,  and  reduced 
defects  by  about  75  percent.  According  to  a  1999  SEI  report,  a  software 
development  contractor  reduced  its  average  deviation  fi:om  estimated 
schedule  time  firom  112  percent  to  5  percent  between  1988  and  1996. 
During  the  same  period,  SEI  reported  that  this  contractor  reduced  its 
average  deviation  from  estimated  cost  from  87  percent  to  minus  4  percent. 


DLA  does  not  currently  have  a  software  process  improvement  program, 
and  recent  efforts  to  establish  one  have  not  made  much  progress.  We 
recently  reported  on  DOD’s  software  process  improvement  efforts, 
including  those  within  DLA.  Specifically,  we  reported  that  before  1998, 
DLA  had  a  software  process  improvement  program^^  however,  DLA 
eliminated  it  during  a  reorganization  in  1998.  In  response  to  our  report, 
DLA’s  Chief  Information  Officer  said  that  the  software  process 
improvement  program  was  to  be  reestablished  during  fiscal  year  2001  and 
that  DLA’s  goal  would  be  for  its  system  developers  and  acquirers  to  reach 
a  level  2  on  the  CMM  by  fiscal  year  2002. 


To  date,  DLA  has  established  an  integrated  product  team  for  software 
process  improvement  that  is  tasked  to  study  DLA’s  software  processes 
and,  based  on  this  study,  to  make  recommendations  on  areas  in  which 
DLA  needs  to  improve.  DLA  has  also  dropped  its  goal  of  achieving  level  2 
by  2002,  and  it  does  not  intend  to  specify  a  CMM  level  for  its  contractors. 
The  software  process  improvement  team  has  produced  several  draft 
papers  and  a  draft  policy,  but  it  does  not  have  a  plan  or  milestones  for 
achieving  software  process  improvement.  According  to  an  agency  official 


“Technical  report  CMU/SEI-95-TR-017,  November  1995. 
^^Technical  Report  CMU/SEI-99-TR-027,  November  1999. 
^^GAO-01-116,  March  30,  2001. 
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associated  with  DLA’s  process  improvement  effort,  funding  to  develop  and 
implement  a  software  process  improvement  program  has  not  been 
approved  because  of  other  agency  IT  funding  priorities,  such  as  BSM. 


Conclusions 


DLA  does  not  have  the  institutional  management  capabilities  necessary  for 
effectively  acquiring  quality  software  repeatedly  on  one  project  after 
another.  This  lack  of  agencywide  consistency  in  software  acquisition 
management  controls  means  that  software  project  success  at  DLA 
currently  depends  more  on  the  individuals  assigned  to  a  given  project  than 
on  the  rules  governing  how  any  assigned  individuals  wiU  function.  That 
has  proven  to  be  a  risky  way  to  manage  softwaredntensive  acquisitions. 

To  DLA’s  benefit,  it  currently  has  a  model  software  acquisition  project 
(BSM)  that,  albeit  not  perfect,  is  a  solid  example  from  which  to  leverage 
lessons  learned  and  replicate  effective  software  acquisition  practices 
across  the  agency.  To  do  so  effectively,  however,  DLA  wiQ  need  to 
implement  a  formal  software  process  improvement  program  and  devote 
adequate  resources  to  correct  the  weaknesses  in  the  software  acquisition 
processes  discussed  in  this  report.  It  wiU  also  have  to  commit  the 
resources  needed  to  implement  a  software  process  improvement  program. 


R  ppnm  TYi  pn  H  ti  nn  q  f nr  reduce  the  software  acquisition  risks  associated  with  its  two  ongoing 

acquisition  projects,  we  recommend  that  the  Secretary  of  Defense  direct 
ExGCUtiV6  Action  the  Director  of  DLA  to  immediately  correct  each  BSM  and  FAS  software- 

acquisition-practice  weakness  identified  in  this  report. 

To  ensure  that  DLA  has  in  place  the  necessary  process  controls  to  acquire 
quality  software  consistently  on  future  acquisition  projects,  we 
recommend  that  the  Secretaiy  also  direct  the  DLA  Director  to 


•  issue  a  policy  requiring  that  (1)  DLA  software-intensive  acquisition 
projects  satisfy  all  applicable  SEl  SA-CMM  level-2  key  process  areas  and 
the  level-3  risk  management  key  process  area  and  (2)  DLA  software 
contractors  have  comparable  software  process  maturity  levels;  and 

•  direct  the  Chief  Information  Officer  (CIO)  to  establish  and  sustain  a 
software  process  improvement  program,  including  (1)  developing  and 
implementing  a  software  process  improvement  plan  that  specifies 
measurable  goals  and  milestones,  (2)  providing  adequate  resources  to  the 
program,  and  (3)  reporting  to  the  Director  every  6  months  on  progress 
against  plans. 
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DOD  provided  what  it  termed  “official  oral  comments”  from  the  Deputy 
Under  Secretary  for  Logistics  and  Materiel  Readiness  on  a  draft  of  this 
report.  In  its  comments,  DOD  stated  that  it  generally  concurred  with  the 
report  and  concurred  with  the  recommendations.  In  particular,  DOD 
stated  that  it  will  issue  policy  directives  requiring  the  Director  of  DLA  to 
(1)  correct  identified  software  acquisition  practice  weaknesses,  except  in 
circumstances  in  which  corrections  to  past  events  make  doing  so 
impractical;  (2)  implement  a  plan  in  all  software-intensive  projects  to 
satisfy  all  applicable  SEI SA-CMM  level-2  and  level-3  key  process  areas, 
and  require  aU  DLA  software  contractors  to  have  comparable  software 
process  maturity  levels;  and  (3)  establish  and  sustain  a  software  process 
improvement  program  that  includes  a  plan  specifying  measurable  goals 
and  milestones,  provides  adequate  resources,  and  reports  to  the  Director 
of  DLA  every  6  months  on  progress  against  the  plan. 


We  are  sending  copies  of  this  report  to  the  Chairmen  and  Ranking 
Minority  Members  of  the  Senate  Appropriations  Subcommittee  on 
Defense;  the  Subcommittee  on  Readiness  and  Management  Support, 
Senate  Committee  on  Armed  Services;  the  House  Appropriations 
Subcoirunittee  on  Defense;  and  the  Subcommittee  on  Readiness,  House 
Committee  on  Armed  Services.  We  are  also  sending  copies  to  the  Director, 
Office  of  Management  and  Budget;  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology;  the  Deputy  Under  Secretary  of  Defense  for 
Logistics  and  Materiel  Readiness;  and  the  Director,  Defense  Logistics 
Agency.  Copies  will  be  made  available  to  others  upon  request. 

If  you  have  any  questions  regarding  this  report,  please  contact  me  at  (202) 
512-3439  or  by  e-mail  at  hiter@gao.gov.  An  additional  GAO  contact  and 
staff  acknowledgements  are  listed  in  appendix  IV. 


Randolph  C.  Hite 

Director,  Information  Technology  Systems  Issues 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


Our  objectives  were  to  determine  (1)  whether  the  Defense  Logistics 
Agency  (DLA)  has  the  effective  software  acquisition  processes  necessary 
to  modernize  and  maintain  systems  and  (2)  what  actions  DLA  has  planned 
or  in  place  to  improve  these  processes. 

To  determine  whether  DLA  has  effective  software  acquisition  processes, 
we  applied  the  Software  Engineering  Institute’s  (SEI)  Software  Acquisition 
Capability  Maturity  Model  using  our  SEI-trained  analysts.  We  focused  on 
the  key  process  areas  necessary  to  obtain  a  repeatable  level  of  maturity, 
the  second  level  of  SEI’s  five-level  model.  We  also  evaluated  against  one 
level-3  key  process  area — acquisition  risk  management — ^because  of  its 
importance.  We  met  with  project  managers  and  project  team  members  to 
determine  whether  and  to  what  extent  they  implemented  each  key 
practice,  and  to  obtain  relevant  documentation.  In  accordance  with  the 
SEI  model,  for  each  key  process  area  we  reviewed, we  evaluated  DLA’s 
institutional  policies  and  practices  and  compared  project-specific 
guidance  and  practices  against  the  required  key  practices. 

More  specifically,  for  each  key  practice  we  reviewed,  we  compared 
project-specific  documentation  and  practices  against  the  criteria  in  the 
software  acquisition  model.  If  the  project  met  the  criteria  for  the  key 
practice  reviewed,  we  rated  it  as  a  strength.  If  the  project  did  not  meet  the 
criteria  for  the  key  practice  reviewed,  we  rated  it  as  a  weakness.  If  the 
evidence  was  mixed  or  inconclusive  and  did  not  support  a  rating  of  either 
a  strength  or  a  weakness,  we  treated  it  as  an  observation.  If  the  key 
practice  was  not  relevant  to  the  project,  we  did  not  rate  it. 

We  evaluated  DLA’s  only  two  software  acquisition  projects  underway  at 
the  time  of  our  review:  the  Business  Systems  Modernization  (BSM)  and  the 
Fuels  Automated  System  (FAS). 

To  determine  what  actions  DLA  has  planned  or  in  place  to  improve  its 
software  processes,  we  identified  the  group  within  DLA  that  is  tasked  with 


evaluated  BSM  in  six  of  the  seven  level-2  key  process  areas — software  acquisition 
planning,  solicitation,  requirements  development  and  management,  project  management, 
contract  tracking  and  oversight,  and  evaluation.  We  evaluated  FAS  in  five  of  the  seven 
level-2  key  process  areas,  as  listed  above,  except  for  solicitation.  We  did  not  evaluate  FAS 
on  solicitation  because  it  is  a  sole-source  procurement.  We  did  not  evaluate  BSM  or  FAS  on 
the  seventh  key  practice  area — transition  to  support— because  the  contractors  who  are 
implementing  these  systems  will  also  support  the  systems  when  they  are  operational, 
rendering  transition  to  support  irrelevant.  We  also  evaluated  BSM  and  FAS  on  one  level-3 
key  process  area — acquisition  risk  management. 


Page  18 


GAO-02-9  DLA's  Acquisition  Maturity 


Appendix  I:  Objectives,  Scope,  and 
Methodology 


performing  this  function.  We  interviewed  agency  officials  who  are 
involved  in  software  process  improvement,  collected  data,  and  analyzed 
draft  policies  and  draft  working  papers  describing  planned  work. 

We  performed  our  work  from  May  through  October  2001,  in  accordance 
with  generally  accepted  government  auditing  standards. 
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Appendix  II:  Results  of  Software  Acquisition 
Capability  Maturity  Model  Evaluation  for 
Business  Systems  Modernization 


Table  3:  Software  Acquisition  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  planning  the  software 
acquisition. 

The  acquisition  organization,  which  is  DLA,  has 
a  written  policy — ^The  Defense  Acquisition 

System  (DODD  5000)— for  planning  the  software 
acquisition. 

Strength® 

Commitment  2 

Responsibility  for  software  acquisition 
planning  activities  is  designated. 

Responsibility  for  software  acquisition  planning 
activities  is  assigned  to  the  BSM  program 
manager. 

Strength 

Ability  1 

A  group  that  is  responsible  for  planning 
the  software  acquisition  exists. 

The  BSM  program  office  is  responsible  for 
planning  the  software  acquisition. 

Strength 

Ability  2 

The  acquisition  organization  provides 
experienced  software  acquisition 
management  personnel  to  support  project 
software  acquisition  planning. 

DLA  provides  experienced  software  acquisition 
management  personnel  to  support  program 
software  acquisition  planning. 

Strength 

Ability  3 

Adequate  resources  are  provided  for 
software  acquisition  planning  activities. 

According  to  BSM  program  officials,  adequate 
resources  are  provided  for  software  acquisition 
planning  activities. 

Strength 

Activity  1 

Software  acquisition  planning  personnel 
are  involved  in  system  acquisition 
planning. 

Software  acquisition  planning  personnel  are 
involved  in  system  acquisition  planning. 

Strength 

Activity  2 

The  project’s  software  acquisition 
planning  is  accomplished  in  conjunction 
with  system  acquisition  planning. 

The  BSM  program’s  software  acquisition 
planning  is  accomplished  in  conjunction  with 
system  acquisition  planning. 

Strength 

Activity  3 

The  software  acquisition  strategy  for  the 
project  is  developed  and  documented. 

The  software  acquisition  strategy  for  the 
program  is  developed  and  documented  in  the 
Acquisition  Strategy  Plan. 

Strength 

Activity  4 

Software  acquisition  planning  addresses 
the  elements  of  the  software  acquisition 
process. 

Software  acquisition  planning  addresses  the 
elements  of  the  software  acquisition  process, 
such  as  program  management,  requirements 
development  and  management,  contract  tracking 
and  oversight,  and  evaluation. 

Strength 

Activity  5 

The  project’s  software  acquisition 
planning  is  documented  and  the  planning 
documentation  is  maintained  over  the  life 
of  the  project. 

The  BSM  program’s  software  acquisition 
planning  is  documented  and  the  planning 
documentation  is  maintained  over  the  life  of  the 
program. 

Strength 

Activity  6 

Life-cycle  support  of  the  software  is 
included  in  software  acquisition  planning 
documentation. 

Life-cycle  support  of  the  software,  such  as 
identifying  adequate  facilities  and  resources  for 
the  software  support  organization,  is  included  in 
software  acauisition  planning  documentation. 

Strength 

Activity  7 

Life-cycle  cost  and  schedule  estimates  for 
the  software  products  and  services  being 
acquired  are  prepared  and  independently 
reviewed. 

Life-cycle  cost  and  schedule  estimates  for  the 
software  products  and  services  being  acquired 
are  prepared  by  the  BSM  program  office  and 
independently  reviewed  by  the  Naval  Center  for 
Cost  Analysis. 

Strength 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  software 
acquisition  planning  activities  and 
resultant  products. 

Measurements,  such  as  metrics  that  track 
software  acquisition  planning  activities  and 
compare  them  to  baselines,  are  made  and  used 
to  determine  the  status  of  the  software 
acquisition  planning  activities  and  resultant 
products. 

Strength 
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Capability  Maturity  Model  Evaluation  for 

Business  Systems  Modernization 

Common  feature 

Key  practice 

Finding 

Rating 

Verification  1 

Software  acquisition  planning  activities 
are  reviewed  by  acquisition  organization 
management  on  a  periodic  basis. 

Software  acquisition  planning  activities  are 
reviewed  by  the  DLA  Executive  Board  on  a 
quarterly  basis. 

Strength 

Verification  2 

Software  acquisition  planning  activities 
are  reviewed  by  the  project  manager  on 
both  a  periodic  and  event-driven  basis. 

Software  acquisition  planning  activities  are 
reviewed  by  the  program  manager  on  both  a 
weekly  and  event-driven  basis. 

Strength 

“Strength  =  Key  practice  effectively  implemented. 


Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Appendix  II;  Results  of  Software  Acquisition 

Capability  Maturity  Model  Evaluation  for 

Business  Systems  Modernization 

Table  4:  Solicitation  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  the  conduct  of  the  software  portion  of 
the  solicitation. 

The  BSM  program  officials  stated  that  The 
Defense  Acquisition  System  (DODD  5000)  is 
the  written  policy  for  the  conduct  of  the  software 
portion  of  the  solicitation;  however,  this  directive 
does  not  address  the  conduct  of  the  software 
portion  of  the  solicitation. 

Weakness® 

Commitment  2 

Responsibility  for  the  software  portion  of  the 
solicitation  is  designated. 

Responsibility  for  the  software  portion  of  the 
solicitation  is  assigned  to  the  Contracting 

Officer. 

Strength 

Commitment  3 

A  selection  official  has  been  designated  to  be 
responsible  for  the  selection  process  and  the 
decision. 

The  Contracting  Officer  has  been  designated 
the  selection  official  responsible  for  the 
selection  process  and  the  decision. 

Strength 

Ability  1 

A  group  that  is  responsible  for  coordinating 
and  conducting  solicitation  activities  exists. 

The  BSM  Acquisition  Integrated  Product  Team 
is  responsible  for  coordinating  and  conducting 
solicitation  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for 
solicitation  activities. 

According  to  BSM  program  officials,  adequate 
resources  are  provided  for  solicitation  activities. 

Strength 

Ability  3 

Individuals  performing  solicitation  activities 
have  experience  or  receive  training. 

Individuals  performing  solicitation  activities  have 
experience  and  receive  training. 

Strength 

Ability  4 

The  groups  supporting  the  solicitation  (e.g., 
end  user,  systems  engineering,  software 
support  organization,  and  application  domain 
experts)  receive  orientation  on  the  solicitation’s 
objectives  and  procedures. 

The  groups  supporting  the  solicitation  (e.g.,  end 
user,  systems  engineering,  software  support 
organization,  and  application  domain  experts) 
receive  orientation  on  the  solicitation’s 
obiectives  and  procedures. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  solicitation 
plans. 

The  BSM  program  office  performs  its  activities 
in  accordance  with  its  documented  solicitation 
plans. 

Strength 

Activity  2 

Solicitation  activities  are  conducted  in  a 
manner  compliant  with  relevant  laws,  policies, 
and  guidance. 

Solicitation  activities  are  conducted  in  a  manner 
compliant  with  relevant  laws,  policies,  and 
guidance. 

Strength 

Activity  3 

The  software  and  evaluation  requirements  are 
incorporated  into  the  solicitation  package  and 
resulting  contract. 

The  software  and  evaluation  requirements  are 
incorporated  Into  the  solicitation  package  and 
resulting  contract. 

Strength 

Activity  4 

Proposals  are  evaluated  in  accordance  with 
documented  solicitation  plans. 

Proposals  are  evaluated  in  accordance  with 
documented  solicitation  plans. 

Strength 

Activity  5 

Cost  and  schedule  estimates  for  the  software 
products  and  services  being  sought  are 
prepared. 

Cost  and  schedule  estimates  for  the  software 
products  and  services  being  sought  are 
prepared. 

Strength 

Activity  6 

Software  cost  and  schedule  estimates  are 
independently  reviewed  for 
comprehensiveness  and  realism. 

Software  cost  and  schedule  estimates  are 
independently  reviewed  by  the  Naval  Center  for 
Cost  Analysis  for  comprehensiveness  and 
realism. 

Strength 

Activity  7 

The  selection  official  uses  proposal  evaluation 
results  to  support  his  or  her  decision  to  select 
an  offeror. 

The  selection  official  uses  proposal  evaluation 
results  to  support  his  decision  to  select  an 
offeror. 

Strength 

Activity  8 

The  project  team  takes  action  to  ensure  the 
mutual  understanding  of  software 
requirements  and  plans  with  the  selected 
offeror{s)  prior  to  contract  signing. 

The  BSM  program  office  takes  actions,  such  as 
meetings,  e-mails,  and  question  and  answer 
sessions,  to  ensure  the  mutual  understanding  of 
software  requirements  and  plans  with  the 
selected  offeror(s)  prior  to  contract  signing. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  solicitation 
activities  and  resultant  products. 

Measurements,  such  as  metrics  that  track 
solicitation  activities  and  compare  them  to 
baselines,  are  made  and  used  to  determine  the 
status  of  the  solicitation  activities  and  resultant 
products. 

Strength 

Verification  1 

Solicitation  activities  are  reviewed  by  the 
acquisition  organization  management  on  a 
periodic  basis. 

Solicitation  activities  are  reviewed  by  the  DLA 
Executive  Board  on  a  quarterly  basis. 

Strength 

Verification  2 

Solicitation  activities  are  reviewed  by  the 
project  manager  or  designated  selection 
official  on  both  a  periodic  and  event-driven 
basis. 

Solicitation  activities  are  reviewed  by  the 
program  manager  and  designated  selection 
official  on  both  a  weekly  and  event-driven  basis. 

Strength 

“Weakness  =  Key  practice  not  effectively  implemented  or  not  implemented. 


Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO, 
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Capability  Maturity  Model  Evaluation  for 

Business  Systems  Modernization 

Table  5:  Requirements  Development  and  Management  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written  policy 
for  establishing  and  managing  the  software- 
related  contractual  requirements. 

The  acquisition  organization,  which  is  DLA,  has  a 
written  policy—The  Defense  Acquisition  System 
(DODD  5000)— for  establishing  and  managing 
the  software-related  contractual  requirements. 

Strength 

Commitment  2 

Responsibility  for  requirements  development 
and  management  is  designated. 

Responsibility  for  requirements  development  and 
management  is  assigned  to  the  BSM  Core 
Integrated  Product  Team. 

Strength 

Ability  1 

A  group  that  is  responsible  for  performing 
requirements  development  and  management 
activities  exists. 

The  BSM  Requirements  Development  and 
Management  Integrated  Product  Team  is 
responsible  for  performing  requirements 
development  and  management  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for 
requirements  development  and  management 
activities. 

According  to  BSM  program  officials,  adequate 
resources  are  provided  for  requirements 
development  and  management  activities. 

Strength 

Ability  3 

Individuals  performing  requirements 
development  and  management  activities  have 
experience  or  receive  training. 

Individuals  performing  requirements  development 
and  management  activities  have  experience  and 
receive  training. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  requirements 
development  and  management  plans. 

The  BSM  program  does  not  have  documented 
requirements  development  and  management 
plans. 

Weakness 

Activity  2 

The  project  team  develops,  baselines,  and 
maintains  software-related  contractual 
requirements  and  places  them  under  change 
control  early  in  the  project,  but  not  later  than 
release  of  the  solicitation  package. 

The  BSM  program  office  team  developed, 
baselined,  and  maintained  software-related 
contractual  requirements  and  placed  them  under 
change  control  at  the  same  time  the  solicitation 
package  was  released. 

Strength 

Activity  3 

The  project  team  appraises  system 
requirements  change  requests  for  their  impact 
on  the  software  being  acquired. 

The  BSM  program  office  does  not  appraise 
system  requirements  change  requests  for  their 
impact  on  the  software  being  acquired. 

Weakness 

Activity  4 

The  project  team  appraises  ail  changes  to  the 
software-related  contractual  requirements  for 
their  Impact  on  performance,  architecture, 
supportability,  system  resource  utilization,  and 
contract  schedule  and  cost. 

The  BSM  program  office  does  not  appraise  all 
changes  to  the  software-related  contractual 
requirements  for  their  impact  on  performance, 
architecture,  supportability,  system  resource 
utilization,  and  contract  schedule  and  cost. 

Weakness 

Activity  5 

Bi-directional  traceability  between  the 
contractual  requirements  and  the  contractor’s 
team  software  work  products  and  services  is 
maintained  throughout  the  effort. 

The  BSM  program  office  has  a  traceability  matrix 
that  it  uses  to  trace  between  the  contractual 
requirements  and  the  contractor’s  team  software 
work  products  and  services.  The  matrix  is 
maintained  throughout  the  effort. 

Strength 

Activity  6 

The  end  user  and  other  affected  groups  are 
Involved  in  the  development  of  all  software- 
related  contractual  requirements  and  any 
subsequent  change  activity. 

The  end  user  and  other  affected  groups,  such  as 
the  program  management  group,  change 
management  group,  and  technical  management 
group,  are  involved  in  the  development  of  all 
software-related  contractual  requirements  and 
anv  subsequent  change  activity. 

Strength 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  requirements 
development  and  management  activities  and 
resultant  products. 

Measurements,  such  as  metrics  that  track 
requirements  development  and  management 
activities  and  compare  them  to  baselines,  are 
made  and  used  to  determine  the  status  of  the 
requirements  development  and  management 
activities  and  resultant  products. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Verification  1 

Requirements  development  and  management 
activities  are  reviewed  by  acquisition 
organization  management  (and  the  contractor) 
on  a  periodic  basis. 

Requirements  development  and  management 
activities  are  reviewed  by  the  DLA  Executive 
Board  on  a  quarterly  basis  and  by  the  contractor 
on  a  weekly  basis. 

Strength 

Verification  2 

Requirements  development  and  management 
activities  are  reviewed  by  the  project  manager 
on  both  a  periodic  and  event-driven  basis. 

Requirements  development  and  management 
activities  are  reviewed  by  the  program  manager 
on  both  a  weekly  and  event-driven  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  6:  Project  Management  Findings  for  BSM 


Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  execution  of  the  software  project. 

The  acquisition  organization,  which  is  DLA,  has 
a  written  policy — ^The  Defense  Acquisition 

System  (DODD  5000)‘-’for  execution  of  the 
software  program. 

Strength 

Commitment  2 

Responsibility  for  project  management  is 
desiqnated. 

Responsibility  for  program  management  is 
assigned  to  the  BSM  program  manager. 

Strength 

Ability  1 

A  team  that  Is  responsible  for  performing  the 
project’s  software  acquisition  management 
activities  exists. 

The  BSM  Program  Management  Office  Is 
responsible  for  performing  the  program’s 
software  acquisition  management  activities. 

Strength 

Ability  2 

Adequate  resources  for  the  project  team  are 
provided  for  the  duration  of  the  software 
acquisition  project. 

According  to  BSM  program  officials,  adequate 
resources  for  the  program  team  are  provided  for 
the  duration  of  the  software  acquisition  program. 

Strength 

Ability  3 

When  project  trade-offs  are  necessary,  the 
project  manager  Is  permitted  to  alter  the 
performance,  cost,  or  schedule  software 
acquisition  baseline. 

When  program  trade-offs  are  necessary,  the 
program  manager  Is  permitted  to  alter  the 
performance,  cost,  or  schedule  software 
acquisition  baseline. 

Strength 

Ability  4 

The  project  team  has  experience  or 
receives  training  in  project  software 
acquisition  management  activities. 

The  BSM  program  office  has  experience  and 
receives  training  in  program  software  acquisition 
management  activities. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  software 
acquisition  management  plans. 

The  BSM  program  office  performs  its  activities  In 
accordance  with  Its  Acquisition  Strategy  Plan. 

Strength 

Activity  2 

The  roles,  responsibilities,  and  authority  for 
the  project  functions  are  documented, 
maintained,  and  communicated  to  affected 
groups. 

The  roles,  responsibilities,  and  authority  for  the 
program  functions  are  documented  in  the 
Acquisition  Strategy  Plan  and  are  maintained 
and  communicated  to  affected  groups. 

Strength 

Activity  3 

The  project  team’s  commitments,  and 
changes  to  commitments,  are 
communicated  to  affected  groups. 

The  BSM  program  office’s  commitments,  and 
changes  to  commitments,  are  communicated  to 
affected  groups  during  weekly  status  meetings. 

Strength 

Activity  4 

The  project  team  tracks  the  risks  associated 
with  cost,  schedule,  resources,  and  the 
technical  aspects  of  the  project. 

The  BSM  program  office  tracks  the  risks 
associated  with  cost,  schedule,  resources,  and 
the  technical  aspects  of  the  program. 

Strength 

Activity  5 

The  project  team  tracks  project  issues, 
status,  execution,  funding,  and  expenditures 
against  project  plans  and  takes  action. 

The  BSM  program  office  tracks  program  issues, 
status,  execution,  funding,  and  expenditures 
against  program  plans  and  takes  action. 

Strength 

Activity  6 

The  project  team  implements  a  corrective 
action  system  for  the  identification, 
recording,  tracking,  and  correction  of 
problems  discovered  during  the  software 
acquisition. 

The  BSM  program  office  implemented  a 
corrective  action  system  for  the  identification, 
recording,  tracking,  and  correction  of  problems 
discovered  during  the  software  acquisition. 

Strength 

Activity  7 

The  project  team  keeps  its  plans  current 
during  the  life  of  the  project  as  replanning 
occurs,  issues  are  resolved,  requirements 
are  changed,  and  new  risks  are  discovered. 

The  BSM  program  office  keeps  its  plans  current 
during  the  life  of  the  program  as  replanning 
occurs,  Issues  are  resolved,  requirements  are 
changed,  and  new  risks  are  discovered. 

Strength 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant 
products. 

Measurements,  such  as  metrics  that  track 
program  management  activities  and  compare 
them  to  baselines,  are  made  and  used  to 
determine  the  status  of  the  program 
management  activities  and  resultant  products. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Verification  1 

Project  management  activities  are  reviewed 
by  acquisition  organization  management  on  a 
periodic  basis. 

Program  management  activities  are  reviewed 
by  the  DLA  Executive  Board  on  a  quarterly 
basis. 

Strength 

Verification  2 

Project  management  activities  are  reviewed 
by  the  project  manager  on  both  a  periodic 
and  event-driven  basis. 

Program  management  activities  are  reviewed 
by  the  program  manager  on  both  a  weekly  and 
event-driven  basis. 

Strength 

Source:  Key  practice  data  from  SEl;  findings  and  ratings  from  GAO. 
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Table  7:  Contract  Tracking  and  Oversight  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written  policy  for 
the  contract  tracking  and  oversight  of  the  contracted 
software  effort. 

The  acquisition  organization,  which  is  DLA,  has 
a  written  policy— The  Defense  Acquisition 
System  (DODD  5000)— for  the  contract 
tracking  and  oversight  of  the  contracted 
software  effort. 

Strength 

Commitment  2 

Responsibility  for  contract  tracking  and  oversight 
activities  is  designated. 

Responsibility  for  contract  tracking  and 
oversight  activities  is  assigned  to  the  Contract 
Management  Office. 

Strength 

Commitment  3 

The  project  team  includes  contracting  specialists  in 
the  execution  of  the  contract. 

The  BSM  program  office  includes  contracting 
specialists  in  the  execution  of  the  contract. 

Strength 

Ability  1 

A  group  that  is  responsible  for  managing  contract 
tracking  and  oversight  activities  exists. 

The  BSM  Acquisition  and  Contract 

Management  Integrated  Product  Team  is 
responsible  for  managing  contract  tracking  and 
oversight  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for  contract 
tracking  and  oversight  activities. 

According  to  BSM  program  officials,  adequate 
resources  are  provided  for  contract  tracking 
and  oversight  activities. 

Strength 

Ability  3 

Individuals  performing  contract  tracking  and 
oversight  activities  have  experience  or  receive 
training. 

Individuals  performing  contract  tracking  and 
oversight  activities  have  experience  and 
receive  training. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  contract  tracking 
and  oversight  plans. 

The  BSM  program  office  performs  Its  activities 
in  accordance  with  Its  documented  contract 
tracking  and  oversight  plans. 

Strength 

Activity  2 

The  project  team  reviews  required  contractor 
software  planning  documents  which,  when 
satisfactory,  are  used  to  oversee  the  contractor 
team’s  software  engineering  effort. 

The  BSM  program  office  reviews  required 
contractor  software  planning  documents  such 
as  the  program  management  plan,  software 
risk  management  plan,  and  subcontract 
management  plan  which,  when  satisfactory,  it 
uses  to  oversee  the  contractor  team’s  software 
engineering  effort. 

Strength 

Activity  3 

The  project  team  conducts  periodic  reviews  and 
interchanges  with  the  contractor  team. 

The  BSM  program  office  conducts  daily 
reviews  and  interchanges  with  the  contractor 
team. 

Strength 

Activity  4 

The  actual  cost  and  schedule  of  the  contractor’s 
software  engineering  effort  are  compared  to 
planned  schedules  and  budgets  and  issues  are 
identified. 

The  actual  cost  and  schedule  of  the 
contractor’s  software  engineering  effort  are 
compared  to  planned  schedules  and  budgets 
and  issues  are  identified. 

Strength 

Activity  5 

The  size,  critical  computer  resources,  and  technical 
activities  associated  with  the  contractor  team’s  work 
products  are  tracked  and  issues  Identified. 

The  size,  critical  computer  resources,  and 
technical  activities  associated  with  the 
contractor  team’s  work  products  are  tracked 
and  issues  Identified. 

Strength 

Activity  6 

The  project  team  reviews  and  tracks  the 
development  of  the  software  engineering 
environment  required  to  provide  life  cycle  support 
for  the  acquired  software  and  issues  are  identified. 

The  BSM  program  office  reviews  and  tracks 
the  development  of  the  software  engineering 
environment  required  to  provide  life  cycle 
support  for  the  acquired  software  and  issues 
are  identified. 

Strength 

Activity  7 

Any  issues  found  by  the  project  team  during 
contract  tracking  and  oversight  are  recorded  in  the 
appropriate  corrective  action  system,  action  taken, 
and  tracked  to  closure. 

Any  issues  found  by  the  BSM  program  office 
during  contract  tracking  and  oversight  are 
recorded  in  the  appropriate  corrective  action 
svstem,  action  taken,  and  tracked  to  closure. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Activity  8 

The  project  team  ensures  that  changes  to  the 
software-related  contractual  requirements  are 
coordinated  with  all  affected  groups  and  individuals, 
such  as  the  contracting  official,  contractor,  and  end 
user. 

The  BSM  program  office  ensures  that  changes 
to  the  software-related  contractual 
requirements  are  coordinated  with  all  affected 
groups  and  individuals,  such  as  the  contracting 
official,  contractor,  and  end  user. 

Strength 

Measurement  1 

Measurements  are  made  and  used  to  determine  the 
status  of  the  contract  tracking  and  oversight 
activities  and  resultant  products. 

Measurements,  such  as  metrics  that  track 
contract  tracking  and  oversight  activities  and 
compare  them  to  baselines,  are  made  and 
used  to  determine  the  status  of  the  contract 
tracking  and  oversight  activities  and  resultant 
products. 

Strength 

Verification  1 

Contract  tracking  and  oversight  activities  are 
reviewed  by  acquisition  organization  management 
on  a  periodic  basis. 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  DLA  Executive  Board  on  a 
quarterly  basis. 

Strength 

Verification  2 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  project  manager  on  both  a  periodic 
and  event-driven  basis. 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  program  manager  on  both  a 
weekly  and  event-driven  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  8:  Evaluation  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  managing  the  evaluation  of  the 
acquired  software  products  and  services. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy — ^The  Defense  Acquisition 
System  (DODD  5000)— for  managing  the 
evaluation  of  the  acquired  software  products 
and  services. 

Strength 

Commitment  2 

Responsibility  for  evaluation  activities  is 
designated. 

Responsibility  for  evaluation  activities  is 
assigned  to  the  BSM  Program  Management 
Office. 

Strength 

Ability  1 

A  group  that  is  responsible  for  planning, 
managing,  and  performing  evaluation 
activities  for  the  project  exists. 

The  BSM  Test  and  Evaluation  Integrated 
Product  Team  is  responsible  for  planning, 
managing,  and  performing  evaluation  activities 
for  the  program. 

Strength 

Ability  2 

Adequate  resources  are  provided  for 
evaluation  activities. 

According  to  BSM  program  officials,  adequate 
resources  are  provided  for  evaluation 
activities. 

Strength 

Ability  3 

Individuals  performing  evaluation  activities 
have  experience  or  receive  training. 

Individuals  performing  evaluation  activities 
have  experience  and  receive  training. 

Strength 

Ability  4 

Members  of  the  project  team  and  groups 
supporting  the  software  acquisition  receive 
orientation  on  the  objectives  of  the 
evaluation  approach. 

Members  of  the  BSM  program  office  stated 
that  they  received  orientation  on  the  objectives 
of  the  evaluation  approach;  however,  they 
could  not  provide  documentation  to  support 
this. 

Observation® 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  evaluation 
plans. 

The  BSM  program  office  performs  its 
activities,  such  as  assessing  technical  risk, 
reviewing  the  integration  approach,  and 
ensuring  that  resources  are  sufficient,  in 
accordance  with  its  documented  evaluation 
plans. 

Strength 

Activity  2 

The  project’s  evaluation  requirements  are 
developed  in  conjunction  with  the 
development  of  the  system  or  software 
technical  requirements. 

The  BSM  program’s  evaluation  requirements 
are  developed  in  conjunction  with  the 
development  of  the  system  technical 
requirements. 

Strength 

Activity  3 

The  project’s  evaluation  activities  are 
planned  to  minimize  duplication  and  take 
advantage  of  all  evaluation  results,  where 
appropriate. 

The  BSM  program’s  evaluation  activities,  as 
stated  in  the  Test  and  Evaluation  Master  Plan, 
are  planned  to  minimize  duplication  and  take 
advantage  of  all  evaluation  results,  where 
appropriate. 

Strength 

Activity  4 

The  project  team  appraises  the  contractor 
team’s  performance  over  the  total  period  of 
the  contract  for  compliance  with 
requirements. 

The  BSM  program  team  appraises  the 
contractor  team’s  performance  over  the  total 
period  of  the  contract  for  compliance  with 
requirements. 

Strength 

Activity  5 

Planned  evaluations  are  performed  on  the 
evolving  software  products  and  services 
prior  to  acceptance  for  operational  use. 

Planned  evaluations  are  performed  on  the 
evolving  software  products  and  services  prior 
to  acceptance  for  operational  use. 

Strength 

Activity  6 

Results  of  the  evaluations  are  analyzed 
and  compared  to  the  contract’s 
requirements  to  establish  an  objective 
basis  to  support  the  decision  to  accept  the 
products  and  services  or  to  take  further 
action. 

Results  of  the  evaluations  are  analyzed  and 
compared  to  the  contract’s  requirements  to 
establish  an  objective  basis  to  support  the 
decision  to  accept  the  products  and  services 
or  to  take  further  action. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

Measurements,  such  as  metrics  that  track 
evaluation  activities  and  compare  them  to 
baselines,  are  made  and  used  to  determine 
the  status  of  the  evaluation  activities  and 
resultant  products. 

Strength 

Verification  1 

Evaluation  activities  are  reviewed  by 
acquisition  organization  management  on  a 
periodic  basis. 

Evaluation  activities  are  reviewed  by  the  DLA 
Executive  Board  on  a  quarterly  basis. 

Strength 

Verification  2 

Evaluation  activities  are  reviewed  by  the 
project  manager  on  both  a  periodic  and 
event-driven  basis. 

Evaluation  activities  are  reviewed  by  the 
program  manager  on  both  a  weekly  and 
event-driven  basis. 

Strength 

“Observation  =  Key  practice  evaluated,  but  the  practice  cannot  be  rated  as  either  a  strength  or  a 
weakness  because  (1)  documentation  was  not  provided  or  (2)  the  practice  was  only  partially 
implemented. 


Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  9:  Acquisition  Risk  Management  Findings  for  BSM 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  the  management  of  software 
acquisition  risk. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy— The  Defense 

Acquisition  System  (DODD  5000) — for  the 
management  of  software  acquisition  risk. 

Strength 

Commitment  2 

Responsibility  for  software  acquisition  risk 
management  activities  is  designated. 

Responsibility  for  software  acquisition  risk 
management  activities  is  assigned  to  the 

Risk  Management  Office. 

Strength 

Ability  1 

A  group  that  is  responsible  for  coordinating 
software  acquisition  risk  management 
activities  exists. 

BSM’s  Risk  and  Issue  Management 

Integrated  Product  Team  is  responsible  for 
coordinating  software  acquisition  risk 
management  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for 
software  acquisition  risk  management 
activities. 

According  to  BSM  program  officials, 
adequate  resources  are  provided  for  software 
acauisition  risk  management  activities. 

Strength 

Ability  3 

Individuals  performing  software  acquisition 
risk  management  activities  have 
experience  or  receive  required  training. 

Individuals  performing  software  acquisition 
risk  management  activities  have  experience 
and  receive  required  training. 

Strength 

Activity  1 

Software  acquisition  risk  management 
activities  are  integrated  into  software 
acquisition  planning. 

Software  acquisition  risk  management 
activities  are  integrated  into  software 
acquisition  planning. 

Strength 

Activity  2 

The  Software  Acquisition  Risk 

Management  Plan  is  developed  in 
accordance  with  the  project’s  defined 
software  acquisition  process. 

The  Acquisition  Risk  Management  Plan  is 
developed  in  accordance  with  the  program’s 
defined  software  acquisition  process. 

Strength 

Activity  3 

The  project  team  performs  its  software 
acquisition  risk  management  activities  in 
accordance  with  its  documented  plans. 

The  BSM  program  office  performs  its 
software  acquisition  risk  management 
activities  in  accordance  with  its  documented 
Acquisition  Risk  Management  Plan. 

Strength 

Activity  4 

The  project  team  encourages  and  rewards 
project-wide  participation  in  the 
identification  and  mitigation  of  risks. 

The  BSM  program  office  encourages  and 
rewards  program-wide  participation  in  the 
identification  and  mitigation  of  risks.  For 
example,  staff  who  identify  risks  are  publicly 
commended  during  weekly  status  meetings. 

Strength 

Activity  5 

Risk  management  is  conducted  as  an 
integral  part  of  the  solicitation,  project 
performance  management,  and  contract 
performance  management  processes. 

Risk  management  is  conducted  as  an 
integral  part  of  the  solicitation,  program 
performance  management,  and  contract 
performance  management  processes. 

Strength 

Activity  6 

Software  acquisition  risks  are  analyzed, 
tracked,  and  controlled  until  mitigated. 

Software  acquisition  risks  are  analyzed, 
tracked,  and  controlled  until  mitigated. 

Strength 

Activity  7 

Project  reviews  include  the  status  of 
identified  risks. 

Program  reviews  include  the  status  of 
identified  risks. 

Strength 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  acquisition  risk 
management  activities  and  resultant 
products. 

Measurements,  such  as  metrics  that  track 
identified  risks  from  discovery  to  mitigation  to 
closure,  are  made  and  used  to  determine  the 
status  of  the  acquisition  risk  management 
activities  and  resultant  products. 

Strength 

Verification  1 

Acquisition  risk  management  activities  are 
reviewed  by  acquisition  organization 
management  on  a  periodic  basis. 

Acquisition  risk  management  activities  are 
reviewed  by  the  DLA  Executive  Board  on  a 
quarterly  basis. 

Strength 
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Common  feature 

Key  practice 

Finding 

Rating 

Verification  2 

Acquisition  risk  management  activities  are 
reviewed  by  the  project  manager  on  both  a 
periodic  and  event-driven  basis. 

Acquisition  risk  management  activities  are 
reviewed  by  the  program  manager  on  both  a 
weekly  and  event-driven  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  10:  Software  Acquisition  Planning  Findings  for  FAS 


Common  feature 

Commitment  1 


Commitment  2 


Ability  1 
Ability  2 


Ability  3 


Activity  1 
Activity  2 


Activity  3 


Activity  4 


Activity  5 


Activity  6 


Activity  7 


Measurement  1 


Verification  1 


Key  practice _ 

The  acquisition  organization  has  a  written 
policy  for  planning  the  software  acquisition. 


Responsibility  for  software  acquisition  planning 
activities  is  designated. 

A  group  that  is  responsible  for  planning  the 

software  acquisition  exists. _ 

The  acquisition  organization  provides 
experienced  software  acquisition  management 
personnel  to  support  project  software 

acquisition  planning. _ 

Adequate  resources  are  provided  for  software 
acquisition  planning  activities. 

Software  acquisition  planning  personnel  are 

involved  in  system  acquisition  planning. _ 

The  project’s  software  acquisition  planning  is 
accomplished  in  conjunction  with  system 

acquisition  planning. _ 

The  software  acquisition  strategy  for  the  project 
is  developed  and  documented. 

Software  acquisition  planning  addresses  the 
elements  of  the  software  acquisition  process. 


The  project’s  software  acquisition  planning  is 
documented  and  the  planning  documentation  is 
maintained  over  the  life  of  the  project. 

Life-cycle  support  of  the  software  Is  included  in 
software  acquisition  planning  documentation. 


Life-cycle  cost  and  schedule  estimates  for  the 
software  products  and  services  being  acquired 
are  prepared  and  Independently  reviewed. 


Measurements  are  made  and  used  to 
determine  the  status  of  the  software  acquisition 
planning  activities  and  resultant  products. 

Software  acquisition  planning  activities  are 
reviewed  by  acquisition  organization 
management  on  a  periodic  basis. 


Finding _ 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy— The  Defense 
Acquisition  System  (DODD  5000)— for 

planning  the  software  acquisition. _ 

Responsibility  for  software  acquisition 
planning  activities  is  assigned  to  the  FAS 

program  manager. _ 

The  FAS  program  office  is  responsible  for 

planning  the  software  acquisition. _ 

DLA  provides  experienced  software 
acquisition  management  personnel  to 
support  program  software  acquisition 

planning. _ 

According  to  FAS  program  officials, 
adequate  resources  are  not  provided  for 
software  acquisition  planning  activities. 
Software  acquisition  planning  personnel  are 
involved  in  system  acquisition  planning. 

The  program’s  software  acquisition  planning 
is  accomplished  In  conjunction  with  system 

acquisition  planning. _ 

The  software  acquisition  strategy  for  the 
program  is  developed  and  documented  in 

the  Acquisition  Strategy  Plan. _ 

Software  acquisition  planning  addresses  the 
elements  of  the  software  acquisition 
process,  such  as  program  management, 
requirements  development  and 
management,  contract  tracking  and 

oversight,  and  evaluation. _ 

The  program’s  software  acquisition  planning 
is  documented;  however,  there  is  no 
evidence  that  the  planning  documentation  is 
maintained  over  the  life  of  the  program. 
Life-cycle  support  of  the  software,  such  as 
identifying  adequate  facilities  and  resources 
for  the  software  support  organization,  are 
included  in  software  acquisition  planning 

documentation. _ 

Life-cycle  cost  and  schedule  estimates  for 
the  software  products  and  services  being 
acquired  are  prepared  and  independently 

reviewed. _ 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  software 
acquisition  planning  activities  and  resultant 

products. _ 

Software  acquisition  planning  activities  are 
reviewed  by  the  DLA  Executive  Board  on  a 
quarterly  basis.  _ 


Rating 

Strength® 


Strength 


Strength 

Strength 


Weakness 


Strength 

Strength 


Strength 


Strength 


Observation*" 


Strength 


Strength 


Weakness 


Strength 
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Common  feature  Key  practice 

Findinq 

Ratinq 

Verification  2  Software  acquisition  planning  activities  are 

reviewed  by  the  project  manager  on  both  a 
periodic  and  event-driven  basis. 

Software  acquisition  planning  activities  are 
reviewed  by  the  program  manager  on  a 
daily  basis. 

Strength 

“Strength  =  Key  practice  effectively  implemented. 


'Weakness  =  Key  practice  not  effectively  implemented  or  not  implemented. 

“Observation  =  Key  practice  evaluated,  but  the  practice  cannot  be  rated  as  either  a  strength  or  a 
weakness  because  (1)  documentation  was  not  provided  or  (2)  the  practice  was  only  partially 
implemented. 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  11 :  Requirements  Development  and  Management  Findings  for  FAS 

Common  features 

Key  Practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  establishing  and  managing  the 
software-related  contractual  requirements. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy— The  Defense 

Acquisition  System  (DODD  5000)— for 
establishing  and  managing  the  software- 
related  contractual  requirements. 

Strength 

Commitment  2 

Responsibility  for  requirements  development 
and  management  is  designated. 

Responsibility  for  requirements  development 
and  management  is  assigned  to  the  FAS 
proqram  manager. 

Strength 

Ability  1 

A  group  that  is  responsible  for  performing 
requirements  development  and  management 
activities  exists. 

The  Product  Assurance  Group  is  responsible 
for  performing  requirements  development 
and  management  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for 
requirements  development  and  management 
activities. 

According  to  FAS  program  officials,  adequate 
resources  are  not  provided  for  requirements 
development  and  management  activities. 

Weakness 

Ability  3 

Individuals  performing  requirements 
development  and  management  activities  have 
experience  or  receive  training. 

FAS  program  officials  said  that  individuals 
performing  requirements  development  and 
management  activities  have  experience  and 
receive  training.  However,  they  could  not 
provide  documents  to  support  this. 

Observation 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  requirements 
development  and  management  plans. 

The  FAS  program  does  not  have 
documented  requirements  development  and 
management  plans. 

Weakness 

Activity  2 

The  project  team  develops,  baselines,  and 
maintains  software-related  contractual 
requirements  and  places  them  under  change 
control  early  in  the  project,  but  not  later  than 
release  of  the  solicitation  package. 

The  FAS  program  office  did  not  develop, 
baseline,  and  maintain  software-related 
contractual  requirements  and  place  them 
under  change  control  before  the  contract  was 
awarded. 

Weakness 

Activity  3 

The  project  team  appraises  system 
requirements  change  requests  for  their  Impact 
on  the  software  being  acquired. 

The  FAS  program  office  does  not  appraise 
system  requirements  change  requests  for 
their  impact  on  the  software  being  acquired. 

Weakness 

Activity  4 

The  project  team  appraises  all  changes  to  the 
software-related  contractual  requirements  for 
their  impact  on  performance,  architecture, 
supportability,  system  resource  utilization,  and 
contract  schedule  and  cost. 

The  FAS  program  office  does  not  appraise 
changes  to  the  software-related  contractual 
requirements  for  their  impact  on 
performance,  architecture,  supportability, 
system  resource  utilization,  and  contract 
schedule  and  cost. 

Weakness 

Activity  5 

Bi-directional  traceability  between  the 
contractual  requirements  and  the  contractor’s 
team  software  work  products  and  services  Is 
maintained  throughout  the  effort. 

The  FAS  program  office  has  a  traceability 
matrix  that  it  uses  to  trace  between  the 
contractual  requirements  and  the  contractor’s 
team  software  work  products  and  services. 
The  matrix  Is  maintained  throughout  the 
effort. 

Strength 

Activity  6 

The  end  user  and  other  affected  groups  are 
involved  in  the  development  of  all  software- 
related  contractual  requirements  and  any 
subsequent  change  activity. 

The  end  user  and  other  affected  groups  are 
involved  in  the  development  of  all  software- 
related  contractual  requirements;  however, 
the  team  could  not  provide  evidence  of  how 
affected  groups  were  involved  in  changes  to 
software  requirements. 

Observation 
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Common  feature 

Key  practice 

Finding 

Rating 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  requirements 
development  and  management  activities  and 
resultant  products. 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  requirements 
development  and  management  activities  and 
resultant  products. 

Weakness 

Verification  1 

Requirements  development  and  management 
activities  are  reviewed  by  acquisition 
organization  management  (and  the  contractor) 
on  a  periodic  basis. 

Requirements  development  and 
management  activities  are  reviewed  by  the 
DLA  Executive  Board  on  a  quarterly  basis. 

Strength 

Verification  2 

Requirements  development  and  management 
activities  are  reviewed  by  the  project  manager 
on  both  a  periodic  and  event-driven  basis. 

Requirements  development  and 
management  activities  are  reviewed  by  the 
program  manager  on  a  daily  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  12:  Project  Management  Findings  for  FAS 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written 
policy  for  execution  of  the  software  project. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy— -The  Defense  Acquisition 
System  (DODD  5000)— for  execution  of  the 
software  program. 

Strength 

Commitment  2 

Responsibility  for  project  management  is 
designated. 

Responsibility  for  program  management  is 
assigned  to  the  FAS  program  manager. 

Strength 

Ability  1 

A  team  that  is  responsible  for  performing  the 
project’s  software  acquisition  management 
activities  exists. 

The  FAS  program  office  is  responsible  for 
performing  the  program’s  software  acquisition 
management  activities. 

Strength 

Ability  2 

Adequate  resources  for  the  project  team  are 
provided  for  the  duration  of  the  software 
acquisition  project. 

According  to  FAS  program  officials,  adequate 
resources  for  the  program  team  are  not 
provided  for  the  duration  of  the  software 
acquisition  program. 

Weakness 

Ability  3 

When  project  trade-offs  are  necessary,  the 
project  manager  is  permitted  to  alter  the 
performance,  cost,  or  schedule  software 
acquisition  baseline. 

When  project  trade-offs  are  necessary,  the 
program  manager  is  permitted  to  alter  the 
performance,  cost,  or  schedule  software 
acquisition  baseline. 

Strength 

Ability  4 

The  project  team  has  experience  or  receives 
training  in  project  software  acquisition 
management  activities. 

The  FAS  program  office  receives  training  in 
program  software  acquisition  management 
activities. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  software 
acquisition  management  plans. 

The  FAS  program  office  performs  its  activities 
in  accordance  with  its  documented  Acquisition 
Strategy  Plan. 

Strength 

Activity  2 

The  roles,  responsibilities,  and  authority  for  the 
project  functions  are  documented,  maintained, 
and  communicated  to  affected  groups. 

The  roles,  responsibilities,  and  authority  for 
the  program  functions  are  not  documented, 
maintained,  and  communicated  to  affected 
groups. 

Weakness 

Activity  3 

The  project  team’s  commitments,  and  changes 
to  commitments,  are  communicated  to  affected 
groups. 

The  FAS  program  office’s  commitments,  and 
changes  to  commitments,  are  communicated 
to  affected  groups  during  weekly  status 
meetings. 

Strength 

Activity  4 

The  project  team  tracks  the  risks  associated 
with  cost,  schedule,  resources,  and  the 
technical  aspects  of  the  project. 

The  FAS  program  office  does  not  track  the 
risks  associated  with  cost,  schedule, 
resources,  and  the  technical  aspects  of  the 
program. 

Weakness 

Activity  5 

The  project  team  tracks  project  issues,  status, 
execution,  funding,  and  expenditures  against 
project  plans  and  takes  action. 

The  FAS  program  office  does  not  track 
program  issues,  status,  execution,  funding, 
and  expenditures  against  program  plans  and 
take  action. 

Weakness 

Activity  6 

The  project  team  implements  a  corrective 
action  system  for  the  identification,  recording, 
tracking,  and  correction  of  problems  discovered 
during  the  software  acquisition. 

The  FAS  program  office  implemented  a 
corrective  action  system  for  the  identification, 
recording,  tracking,  and  correction  of  problems 
discovered  during  the  software  acquisition. 

Strength 

Activity  7 

The  project  team  keeps  its  plans  current  during 
the  life  of  the  project  as  replanning  occurs, 
issues  are  resolved,  requirements  are  changed, 
and  new  risks  are  discovered. 

The  FAS  program  office  has  not  kept  its  plans 
current  during  the  life  of  the  program. 

Weakness 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  project 
management  activities  and  resultant  products. 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  program 
management  activities  and  resultant  products. 

Weakness 
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Common  feature 

Key  practice 

Finding 

Rating 

Verification  1 

Project  management  activities  are  reviewed  by 
acquisition  organization  management  on  a 
periodic  basis. 

Program  management  activities  are  reviewed 
by  the  DLA  Executive  Board  on  a  quarterly 
basis. 

Strength 

Verification  2 

Project  management  activities  are  reviewed  by 
the  project  manager  on  both  a  periodic  and 
event-driven  basis. 

Program  management  activities  are  reviewed 
by  the  program  manager  on  a  daily  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Table  13:  Contract  Tracking  and  Oversight  Findings  for  FAS 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written  policy 
for  the  contract  tracking  and  oversight  of  the 
contracted  software  effort. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy—The  Defense 

Acquisition  System  (DODD  5000) — ^for  the 
contract  tracking  and  oversight  of  the 
contracted  software  effort. 

Strength 

Commitment  2 

Responsibility  for  contract  tracking  and  oversight 
activities  is  designated. 

Responsibility  for  contract  tracking  and 
oversight  activities  is  assigned  to  the 
contracting  officer’s  technical  representative. 

Strength 

Commitment  3 

The  project  team  includes  contracting  specialists 
in  the  execution  of  the  contract. 

The  FAS  program  office  includes  contracting 
specialists  In  the  execution  of  the  contract. 

Strength 

Ability  1 

A  group  that  Is  responsible  for  managing  contract 
tracking  and  oversight  activities  exists. 

The  FAS  program  office  is  responsible  for 
managing  contract  tracking  and  oversight 
activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for  contract 
tracking  and  oversight  activities. 

According  to  FAS  program  officials,  adequate 
resources  are  not  provided  for  contract 
tracking  and  oversight  activities. 

Weakness 

Ability  3 

Individuals  performing  contract  tracking  and 
oversight  activities  have  experience  or  receive 
training. 

Individuals  performing  contract  tracking  and 
oversight  activities  have  experience. 

Strength 

Activity  1 

The  project  team  performs  its  activities  in 
accordance  with  its  documented  contract  tracking 
and  oversight  plans. 

The  FAS  program  office  does  not  have  a 
contract  tracking  and  oversight  plan. 

Weakness 

Activity  2 

The  project  team  reviews  required  contractor 
software  planning  documents  which,  when 
satisfactory,  are  used  to  oversee  the  contractor 
team’s  software  engineering  effort. 

Although  FAS  program  officials  indicate  that 
they  review  many  of  the  program’s  planning 
documents,  they  could  not  provide  evidence 
that  these  reviews  take  place. 

Observation 

Activity  3 

The  project  team  conducts  periodic  reviews  and 
interchanges  with  the  contractor  team. 

FAS  program  team  conducts  periodic  reviews 
and  interchanges  with  the  contractor  team. 

Strength 

Activity  4 

The  actual  cost  and  schedule  of  the  contractor’s 
software  engineering  effort  are  compared  to 
planned  schedules  and  budgets  and  issues  are 
identified. 

The  actual  cost  and  schedule  of  the 
contractor’s  software  engineering  effort  are 
not  compared  to  planned  schedules  and 
budgets  and  issues  are  not  identified. 

Weakness 

Activity  5 

The  size,  critical  computer  resources,  and 
technical  activities  associated  with  the  contractor 
team’s  work  products  are  tracked,  and  issues 
identified. 

The  size,  critical  computer  resources,  and 
technical  activities  associated  with  the 
contractor  team’s  work  products  are  tracked, 
and  issues  identified. 

Strength 

Activity  6 

The  project  team  reviews  and  tracks  the 
development  of  the  software  engineering 
environment  required  to  provide  life-cycle  support 
for  the  acquired  software  and  issues  are 
identified. 

The  FAS  program  office  reviews  and  tracks 
the  development  of  the  software  engineering 
environment  required  to  provide  life-cycle 
support  for  the  acquired  software  and  issues 
are  identified. 

Strength 

Activity  7 

Any  issues  found  by  the  project  team  during 
contract  tracking  and  oversight  are  recorded  in  the 
appropriate  corrective  action  system,  action  taken, 
and  tracked  to  closure. 

Issues  found  by  the  project  team  during 
contract  tracking  and  oversight  are  recorded 
in  the  appropriate  corrective  action  system, 
action  taken,  and  tracked  to  closure. 

Strength 

Activity  8 

The  project  team  ensures  that  changes  to  the 
software-related  contractual  requirements  are 
coordinated  with  all  affected  groups  and 
individuals,  such  as  the  contracting  official, 
contractor,  and  end  user. 

The  program  team  does  not  ensure  that 
changes  to  the  software-related  contractual 
requirements  are  coordinated  with  all  affected 
groups  and  Individuals,  such  as  the 
contracting  official,  contractor,  and  end  user. 

Weakness 
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Common  feature 

Key  practice 

Finding 

Rating 

Measurement  1 

Measurements  are  made  and  used  to  determine 
the  status  of  the  contract  tracking  and  oversight 
activities  and  resultant  products. 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  contract  tracking 
and  oversight  activities  and  resultant 
products. 

Weakness 

Verification  1 

Contract  tracking  and  oversight  activities  are 
reviewed  by  acquisition  organization  management 
on  a  periodic  basis. 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  DLA  Executive  Board  on  a 
quarterly  basis. 

Strength 

Verification  2 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  project  manager  on  both  a 
periodic  and  event-driven  basis. 

Contract  tracking  and  oversight  activities  are 
reviewed  by  the  program  manager  on  a  daily 
basis. 

Strength 

Source:  Key  practice  data  from  SEl;  findings  and  ratings  from  GAO. 
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Table  14:  Evaluation  Findings  for  FAS 

Common  feature 

Key  practice 

Finding 

Rating 

Commitment  1 

The  acquisition  organization  has  a  written  policy 
for  managing  the  evaluation  of  the  acquired 
software  products  and  services. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy — ^The  Defense 

Acquisition  System  (DODD  5000) — ^for 
managing  the  evaluation  of  the  acquired 
software  products  and  services. 

Strength 

Commitment  2 

Responsibility  for  evaluation  activities  is 
designated. 

Responsibility  for  evaluation  activities  is 
assigned  to  the  FAS  Product  Assurance 

Office. 

Strength 

Ability  1 

A  group  that  is  responsible  for  planning, 
managing,  and  performing  evaluation  activities  for 
the  project  exists. 

The  FAS  Working  Level  Test  and  Evaluation 
Integrated  Product  Team  is  responsible  for 
planning,  managing,  and  performing 
evaluation  activities  for  the  program. 

Strength 

Ability  2 

Adequate  resources  are  provided  for  evaluation 
activities. 

According  to  FAS  program  officials,  adequate 
resources  are  not  provided  for  evaluation 
activities. 

Weakness 

Ability  3 

Individuals  performing  evaluation  activities  have 
experience  or  receive  training. 

Although  FAS  program  officials  said 
individuals  performing  evaluation  activities 
have  experience  or  receive  training,  they 
could  not  provide  documents  to  support  this. 

Observation 

Ability  4 

Members  of  the  project  team  and  groups 
supporting  the  software  acquisition  receive 
orientation  on  the  objectives  of  the  evaluation 
approach. 

Members  of  the  program  team  and  groups 
supporting  the  software  acquisition  received 
orientation  on  the  objectives  of  the  evaluation 
approach. 

Strength 

Activity  1 

The  project  team  performs  Its  activities  in 
accordance  with  its  documented  evaluation  plans. 

The  FAS  program  office  performs  its  activities 
in  accordance  with  its  Testing  and  Evaluation 
Master  Plan. 

Strength 

Activity  2 

The  project’s  evaluation  requirements  are 
developed  in  conjunction  with  the  development  of 
the  system  or  software  technical  requirements. 

The  FAS  program’s  evaluation  requirements 
were  developed  in  conjunction  with  the 
development  of  the  system  technical 
requirements. 

Strength 

Activity  3 

The  project’s  evaluation  activities  are  planned  to 
minimize  duplication  and  take  advantage  of  all 
evaluation  results,  where  appropriate. 

The  FAS  program’s  evaluation  activities,  as 
stated  in  the  Testing  and  Evaluation  Master 
Plan,  are  planned  to  minimize  duplication  and 
take  advantage  of  all  evaluation  results, 
where  appropriate. 

Strength 

Activity  4 

The  project  team  appraises  the  contractor  team’s 
performance  over  the  total  period  of  the  contract 
for  compliance  with  requirements. 

FAS  program  officials  said  that  they  appraise 
the  contractor  team’s  performance  over  the 
total  period  of  the  contract  for  compliance 
with  requirements.  However,  they  could  not 
provide  evidence  to  support  this. 

Observation 

Activity  5 

Planned  evaluations  are  performed  on  the 
evolving  software  products  and  services  prior  to 
acceptance  for  operational  use. 

The  FAS  program  office  plans  to  perform 
evaluations  prior  to  operational  use. 

Not  rated 

Activity  6 

Results  of  the  evaluations  are  analyzed  and 
compared  with  the  contract’s  requirements  to 
establish  an  objective  basis  to  support  the 
decision  to  accept  the  products  and  services  or  to 
take  further  action. 

The  FAS  program  office  has  done  some 
evaluations  and  will  finish  in  August  2001 .  At 
that  time,  the  results  of  the  evaluations  will  be 
analyzed  and  compared  with  the  contract’s 
requirements  to  establish  an  objective  basis 
to  support  the  decision  to  accept  the  products 
and  services  or  to  take  further  action. 

Not  rated 
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Common  feature 

Key  practice 

Finding 

Rating 

Measurement  1 

Measurements  are  made  and  used  to  determine 
the  status  of  the  evaluation  activities  and  resultant 
products. 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  evaluation 
activities  and  resultant  products. 

Weakness 

Verification  1 

Evaluation  activities  are  reviewed  by  acquisition 
organization  management  on  a  periodic  basis. 

Evaluation  activities  are  reviewed  by  the  DLA 
Executive  Board  on  a  quarterly  basis. 

strength 

Verification  2 

Evaluation  activities  are  reviewed  by  the  project 
manager  on  both  a  periodic  and  event-driven 
basis. 

Evaluation  activities  are  reviewed  by  the 
program  manager  on  a  daily  basis. 

Strength 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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Appendix  III:  Results  of  Software  Acquisition 

Capability  Maturity  Model  Evaluation  for 

Fuels  Automated  System 

Table  15:  Acquisition  Risk  Management  Findings  for  FAS 

Common  feature 

Key  practice 

Finding 

Ratinq 

Commitment  1 

The  acquisition  organization  has  a  written  policy 
for  the  management  of  software  acquisition  risk. 

The  acquisition  organization,  which  is  DLA, 
has  a  written  policy — The  Defense  Acquisition 
System  (DODD  5000)— for  the  management 
of  software  acquisition  risk. 

Strength 

Commitment  2 

Responsibility  for  software  acquisition  risk 
management  activities  is  designated. 

Responsibility  for  software  acquisition  risk 
management  activities  is  assigned  to  the  FAS 
program  office. 

Strength 

Ability  1 

A  group  that  is  responsible  for  coordinating 
software  acquisition  risk  management  activities 
exists. 

The  Risk  Review  Board  is  responsible  for 
coordinating  software  acquisition  risk 
management  activities. 

Strength 

Ability  2 

Adequate  resources  are  provided  for  software 
acquisition  risk  management  activities. 

According  to  FAS  program  officials,  adequate 
resources  are  not  provided  for  software 
acquisition  risk  management  activities. 

Weakness 

Ability  3 

Individuals  performing  software  acquisition  risk 
management  activities  have  experience  or 
receive  required  training. 

The  FAS  program  office  stated  that  individuals 
performing  acquisition  risk  management 
activities  have  experience;  however,  they 
could  not  provide  us  with  evidence. 

Observation 

Activity  1 

Software  acquisition  risk  management  activities 
are  integrated  into  software  acquisition 
planning. 

Software  acquisition  risk  management 
activities  are  not  integrated  into  software 
acquisition  planning. 

Weakness 

Activity  2 

The  Software  Acquisition  Risk  Management 

Plan  is  developed  in  accordance  with  the 
project’s  defined  software  acquisition  process. 

The  Software  Acquisition  Risk  Management 
Plan  was  not  developed  in  accordance  with 
the  program’s  defined  software  acquisition 
process. 

Weakness 

Activity  3 

The  project  team  performs  its  software 
acquisition  risk  management  activities  in 
accordance  with  its  documented  plans. 

The  FAS  program  office  does  not  perform 
software  acquisition  risk  management 
activities. 

Weakness 

Activity  4 

The  project  team  encourages  and  rewards 
project-wide  participation  in  the  identification 
and  mitigation  of  risks. 

The  FAS  program  office  does  not  encourage 
and  reward  program-wide  participation  in  the 
identification  and  mitigation  of  risks. 

Weakness 

Activity  5 

Risk  management  is  conducted  as  an  Integral 
part  of  the  solicitation,  project  performance 
management,  and  contract  performance 
management  processes. 

Risk  management  is  not  conducted  as  an 
integral  part  of  the  solicitation,  program 
performance  management,  and  contract 
performance  management  process. 

Weakness 

Activity  6 

Software  acquisition  risks  are  analyzed, 
tracked,  and  controlled  until  mitigated. 

Software  acquisition  risks  are  not  analyzed, 
tracked,  and  controlled  until  mitigated. 

Weakness 

Activity  7 

Project  reviews  include  the  status  of  identified 
risks. 

Meeting  minutes  of  program  reviews  do  not 
include  the  status  of  identified  risks. 

Weakness 

Measurement  1 

Measurements  are  made  and  used  to 
determine  the  status  of  the  acquisition  risk 
management  activities  and  resultant  products. 

Measurements  are  not  made  and  used  to 
determine  the  status  of  the  acquisition  risk 
management  activities  and  resultant  products. 

Weakness 

Verification  1 

Acquisition  risk  management  activities  are 
reviewed  by  acquisition  organization 
management  on  a  periodic  basis. 

Acquisition  risk  management  activities  are  not 
reviewed  by  acquisition  organization 
management. 

Weakness 

Verification  2 

Acquisition  risk  management  activities  are 
reviewed  by  the  project  manager  on  both  a 
periodic  and  event-driven  basis. 

Acquisition  risk  management  activities  are  not 
reviewed  by  the  program  manager. 

Weakness 

Source:  Key  practice  data  from  SEI;  findings  and  ratings  from  GAO. 
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